Memos From the Cube

As a (hobbyist) engineer, I routinely find myself mad at Microsoft. Not because of their products (I personally find the new Office and Vista to be quite good), but because of their crud-tastic approach to engineering.

Microsoft seem happy to define their own specifications (and try to force/coerce others into adopting them), but seem totally unable to follow others. Sometimes even their own specifications are poorly implemented - like the recent OOXML debacle.

Recently I’ve been bitten by their lack of engineering abilities four times. Once, from trying to assist someone into combining Mass Storage and CDC classes into a single device. It turns out Windows won’t allow that because of the way it is built; compound devices are enumerated an interface at a time, and there’s no way to bind more than one interface in a compound device to a single OS supplied driver. At least there’s a hacky way around it, at the expense of compatibility with every other OS on the planet.

Second, through the CDC Ethernet. Admittedly there are two standards regarding this (one newer that the other, after the original was found lacking in several areas), but Microsoft went and screwed everyone over by inventing their own psudo-standard called RNDIS. Want Ethernet over USB? You’ll be writing custom drivers to get it working cross-platform.

Thirdly, their %$#@$ING STUPID XBOX Wireless design. This is a total WTF on several accounts:

1) The dongle has four fixed endpoint pairs, one for each possible paired controller. Why do that, rather than having a single endpoint pair and using some sort of index to distingush the controller to interface with?

2) The entire device and each interface is marked as a proprietary class. One would think that it’s got some super-secret, amazing protocol running underneath it to account for the custom class. Nope - one pair of the endpoints at least are just the USB HID class obfuscated to require custom drivers.

3) Despite being partly a HID device (just obfuscated so that it requires a damn custom driver), it lacks an actual HID report. It appears the returned data is supposed to be interpreted via a known, fixed layout.

They could have circumvented the whole thing by making it a compound device containing:

1) A single USB HID interface, with report descriptors indicating the button layout for the controller. Use report IDs to distinguish between different controllers. That would make the dongle enumerate on any modern OS as a bunch of standard gamepad controllers, no driver required. All the buttons, joysticks and LEDs could then be controlled via the standard HID API.

2) A single USB PID interface, to manage the force feedback. That would allow modern OSes to expose the force feedback capabilities to any modern game, no driver required.

And that’s it. No proprietary drivers, no obfuscation layers, and it would be cross-platform. Of course, any other features which are XBOX proprietary (such as memory cards, etc.) could use a legitimately proprietary additional interface in the device, but the other standard features would still be usable.

Why do they spend *extra* time to re-implement an existing standard, so that it requires more work (for them and others) to use when it has no benefit? Seems like a waste of time and money to me.

My apologies for the downtime yesterday. Haven’t a clue what caused it — looks like a problem with the server provider. Whatever the reason, everything is once again working as it should.

I’ve given my blog a little bit of a face lift via a new WordPress theme. I actually stumbled upon it while looking for a theme for the comical “Glacial Pace” blog my University friends are planning on starting up (based around my friend’s notoriously slow eating pace). While I can’t be certain, the new theme carries distinct similarities to the old, so it probably a newer version by the theme author. In any case, it spruces things up slightly without radically altering the design of the blog so I’ve switched to it.

Based on the few comments to my last entry plus research of my own, I’m thinking of leaving the Ethernet-Over-USB project for a little while. I’ve re-opened my unreleased “StillImageHost” demo code, which will eventually allow PIMA-compatible USB cameras (that’s anything which enumerates as a camera device rather than a Mass Storage device under Windows) to be controlled via the USB AVRs. I’ve already got a few of the commands working, so expect more progress on that soon.

Well the Network ‘08 dinner turned out to be a blast, with lots of interesting people to talk to. Turns out that the guy who invited me, Michael, is a really nice and dynamic guy in person - not that I suspected anything to the contrary. Despite the whole thing looking like some sort of weird pyramid scheme at first glance, the venue was quite nice and I was able to learn a little about the current state of the job market for engineers.

Five minutes ago I managed to get my MIDI demo working. I eventually settled on the “MIDI-OX” software (free for non-commercial use) although any MIDI application capable of MIDI-IN is suitable. After much swearing over the normal MIDI specification I realized that I’d missed a crucial part of the USB-MIDI supplement specification; since the USB protocol can carry many, many MIDI streams, each standard MIDI message has to be prefixed by a byte indicating the virtual cable number and message type. All now works well, and the new demo will join the new HID parser in the next MyUSB library release.

Reading the AVRFreaks.net forums too often can have an unintended side effect - more work. No, I’m not talking about the work of helping others, but rather the work induced by the thoughts implanted by others. A thread about MIDI had me thinking for a little too long, and so I decided to investigate the USB-MIDI class specification.

A little reading later and I’ve found that it should be relatively simple, due to the fact that the USB-MIDI class is built on top of the existing Audio class, which I’ve already experimented with. The only tasks that need to be done it seems is to add in the MIDI IN and OUT terminals to the descriptors, and then the actual MIDI note writing to the appropriate endpoint.

I think it’ll be my next addition to the library - a simple MIDI device which playes notes depending on the joystick position. My only problem is the host-side application; can anyone recommend a freeware application which can play notes sent to the host via MIDI?

I’ll be at the The IET Networking Dinner event tomorrow night, so if any of you are also attending, I’ll see you there!

I’ve caved. I’ve started to write my own book on USB communications, focusing on the Atmel USB AVR line of microcontrollers. No idea how it’ll turn out, but it should be fun either way. The end result should be easy enough for competent AVR programmers to use - since the USBXXX AVRs cover the physical USB layer transparently, the book can skip over all that to reduce confusion. I’ve posted a thread on AVRFreaks about the book outline, so post in there if you want your voice heard!

I got an email from a Stefan Salewski PHD (read: much smarter guy than I) who informed me about his own GPL USBXXX driver. Check it out here. It lacks a lot of the features that MyUSB has, but produces much tighter code due to being a bare-bones approach.

Yesterday I received my brand new RZRAVEN boards, courtesy of Atmel. Unfortunately, like so many others, my units came with broken speakers; good thing Atmel says they’ll be working on ways to prevent such problems when the kits start shipping. No firmware available for the boards yet, so I’m restricted to playing with the default firmware. I’ve already added the RZUSBSTICK board to the MyUSB library, which was a very easy task due to the lack of peripherals on the board and the fact that it’s based on the AT90USB1287 AVR.

I’m working on MyUSB 1.3.1, which will be a minor point release to fix a few minor issues in the 1.3.0 release. The demos will be patched so that more of them work on the AT90USBXXX2 AVRs - the 1.3.0 release has several demos which either use more USB FIFO memory than available or used endpoints which don’t support double banking on the smaller reduced device USB AVRs. Also added are some some new Endpoint and Pipe stream read/write functions, which provide a safe API for the exchange of a set amount of bytes stored in a buffer.

I’ll include the unfinished bootloader, just in case anyone else feels like debugging the remaining flash write code and gets it working.

I’m happy to say that my two month old Toshiba Satellite A200/02H now works flawlessly. After one trip to the repair center and a new motherboard later, I was still having the same random “black screen of death” problems which plague the Satellite series laptops. However, a few days ago Toshiba posted a fix in the form of a BIOS update which resolves the issue. Huzzah!

There’s been a bit of confusion concerning the USB Device Descriptors, so I thought I’d do another informational post to clear things up a bit.

All USB devices contain descriptors. These descriptors describe the important aspects of the device to the host, and are read during enumeration so that the appropriate drivers can be loaded for each connected device. There are many types of descriptors, but for basic devices only a few are used; Device Descriptors, Configuration Descriptors, Interface Descriptors and Endpoint Descriptors.

Let’s consider a canned example, that of a USB still camera. Such a camera might have two different USB connection modes; Mass Storage (for drag-drop access like a USB flash disk) and Still Image (for use with supporting camera software). It might also allow the host to use the microphone on the camera, and use the camera element as a low resolution webcam.

We’ll now be designing the descriptors required for such a device, starting with the Device Descriptor.

Device Descriptor
The Device Descriptor is mandatory, and only one is allowed for each USB device. It contains important information about the device, including the manufacturer/product IDs (assigned via USB.org, and provide a unique combination which can identify a device and indicate driver compatibility), Class and Protocol values, total configurations and power requirements.

Some devices (e.g. most Mice and Keyboards) are pretty simple, and only provide one function. These sorts of devices can declare their overall function in the device descriptor’s Class/Protocol values - which should be sourced off the list of defined classes at USB.org - which the host can read. For more complex devices which do not belong to one clear class, these are set as 0×00, which indicates to the host that the classes should be sourced from the interface descriptors of the current configuration.

Descriptors So Far:

• Device Descriptor

Configuration Descriptor
USB devices may have multiple configurations, although these are uncommonly used in practice. Each configuration descriptor completely defines a set of interfaces for the available functions. Configurations are mutually exclusive to one-another; the host may select any one of the configurations at any one time, but cannot use multiple configurations at once.

The Configuration Descriptor contains a set of Interface and Endpoint descriptors, in a flat structure. The actual layout of the configuration descriptor is hierarchical but when transmitted to the host it is sent as a flat datastream.

Our USB camera would have two configurations, one with the Video (camera), Audio (microphone) and Mass Storage features, and another containing Video (camera), Audio (microphone) and Still Image features. As the configurations are separate the common features between configurations need to be repeated inside each independent configuration descriptor.

Descriptors So Far:

• Device Descriptor
• Configuration Descriptor (0)
• Configuration Descriptor (1)

Interface Descriptor
Each device function has one or more interface descriptors associated with it. For example, with our hypothetical USB camera, we would have one interface for the Mass Storage (or Still Image) feature, another for the Video (camera) feature and a third for the Audio (microphone). The interface descriptors indicate the interface’s class and protocol and number of associated endpoints. One other interesting feature of the Interface Descriptors is the Alternate Setting value; like the Configuration descriptor each interface can have multiple entries which can change the interface’s function depending on which one is selected.

Our USB camera will have an alternate setting interface for the audio class; even if a device’s features is not selected, bandwidth is still used by the host for checking the interface’s endpoints. Thus, for audio interfaces (which use up a lot of data), an alternate setting with no endpoints is added so that when the microphone is unused, the host can turn off the endpoint and save on USB bandwidth.

Each interface descriptor will have its Class and Protocol values filled out with the values defined at USB.org for the appropriate functions of each interface.

We’ll assume that the Mass Storage/Still Image class requires two endpoints, and the Video class uses a single endpoint for the purposes of this demonstration.

Descriptors So Far:

• Device Descriptor
• Configuration Descriptor (0)
  • Interface Descriptor (Mass Storage Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
  • Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

• Configuration Descriptor (1)
  • Interface Descriptor (Still Image Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
  • Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

Endpoint Descriptor

Lastly for our simple device, we have the Endpoint Descriptor. Each interface will have 0 or more endpoints, each of which allow for the exchange of data between the device and the host. Each endpoint descriptor indicates the type of endpoint (BULK, CONTROL, INTERRUPT, ISOCHRONOUS), address and (for INTERRUPT endpoints) the polling interval between endpoint interrupts.

Each endpoint must be given a unique address. The value of each address is actually not important, except for the most significant bit, which when set indicates to the host that the endpoint is of the IN type, for sending data from the device into the host. Address 0 is reserved, and is used for the control endpoint that is present on all USB devices for host-device control requests. Endpoint 0 does not need an explicit Endpoint Descriptor — it is mandatory and thus does not need to be described.

We must place each endpoint after the interface to which it belongs. This is how interface-endpoint relationships are established; if an interface declares it has three endpoints, three Endpoint Descriptors must be placed directly after the Interface Descriptor. This is why the address is not important to anything other than the device (which then uses the address to read data in and out of the appropriate endpoints).

Descriptors So Far:

• Device Descriptor
• Configuration Descriptor (0)
  • Interface Descriptor (Mass Storage Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
    • Endpoint Descriptor (BULK, OUT, Address 1, 64 Bytes)
    • Endpoint Descriptor (BULK, IN, Address 2, 64 Bytes)
  • Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
    • Endpoint Descriptor (BULK, IN, Address 3, 64 Bytes)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
    • Endpoint Descriptor (ISOCHRONOUS, IN, Address 4, 128 Bytes)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

• Configuration Descriptor (1)
  • Interface Descriptor (Still Image Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
    • Endpoint Descriptor (BULK, OUT, Address 1, 64 Bytes)
    • Endpoint Descriptor (BULK, IN, Address 2, 64 Bytes)
  • Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
    • Endpoint Descriptor (BULK, IN, Address 3, 64 Bytes)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
    • Endpoint Descriptor (ISOCHRONOUS, IN, Address 4, 128 Bytes)
  • Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

Putting them all together

Now we have the basic hierarchical structure of our descriptors for our USB camera. We need to flatten them into three structures - one Device, two Configuration. To do that, we just add the items to the struct in the order of the above, with children following parents:

Device Descriptor

Configuration Descriptor (0)

Interface Descriptor (Mass Storage Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
Endpoint Descriptor (BULK, OUT, Address 1, 64 Bytes)
Endpoint Descriptor (BULK, IN, Address 2, 64 Bytes)
Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
Endpoint Descriptor (BULK, IN, Address 3, 64 Bytes)
Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
Endpoint Descriptor (ISOCHRONOUS, IN, Address 4, 128 Bytes)
Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

Configuration Descriptor (1)

Interface Descriptor (Still Image Class, Alternate Settings 0, Alternate Setting 0, 2 Endpoints)
Endpoint Descriptor (BULK, OUT, Address 1, 64 Bytes)
Endpoint Descriptor (BULK, IN, Address 2, 64 Bytes)
Interface Descriptor (Video Class, Alternate Settings 0, Alternate Setting 0, 1 Endpoint)
Endpoint Descriptor (BULK, IN, Address 3, 64 Bytes)
Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 0, 1 Endpoint)
Endpoint Descriptor (ISOCHRONOUS, IN, Address 4, 128 Bytes)
Interface Descriptor (Audio Class, Alternate Settings 1, Alternate Setting 1, 0 Endpoints)

Nothing to it. Of course, there are plenty of details about descriptors not covered here, but I just wanted to clear up the general confusion about what to change to create the desired behaviour.

Plenty of people have been asking around on AVRFreaks for information on making a device which is capable of showing up as a removable storage device under Explorer, so that images or logs can be downloaded with any modern operating system. What these people do not know is that the creation of a device is quite a complex task, involving several “layers” which must be correctly implemented before it will work correctly.

I’m taking a break from my usual boring blogs to try to make the inner workings of such a device a little less of a mystery.

First, let’s look at the six different layers required for a typical mass storage device:

The MyUSB Mass Storage Device demo only implements layers 1 through 4. This is because the demo is for a “dumb” storage device, in that all it does is expose a readable/writable storage medium for the host OS. The host’s OS is free to use whatever storage scheme it sees fit to format the device, which is why the demo disk can be formatted to any scheme desired by the user. The device itself never tries to create files on the disk, so the last two layers are unnecessary.

Now, let’s look at the different layers in a little more detail.

LAYER 1 - PHYSICAL LAYER:

The physical layer is the lowest level layer of USB, and is the protocol dictating the exchange of data between a USB host, and one or more devices present on the USB bus. It provides a means of detecting when devices are added or removed, and for data transmission to an addressed device. The AT90USB series AVR microcontrollers include a full USB controller, which takes care of the entire layer 1 for us, allowing us to interact on the physical layer via a group of exposed USB control and status registers.

The physical layer is generally not of any concern to embedded developers, unless of course it is being bit-banged or otherwise generated (as is the case with the software-only USB implementations for some AVR models available on the web) without the aid of a dedicated USB controller.

LAYER 2 - CONTROL LAYER:

Layer 2 is the layer which handles all the device-level requests. This layer controls the creation and modification of the endpoints in a device, the processing of device control commands - including class-specific control commands - and the fetching of the device’s “descriptors” (structures containing information for the host about the device’s capabilities, features, implemented classes and power requirements) for the host

MyUSB handles the control layer transparently, only handing of the processing of such requests if they are class-specific commands rather than general device commands. In the case of the Mass Storage class, there are special control commands for the resetting of the storage device, and for the retrieval of the number of LUNs (Logical UNits, logical disks in the device).

LAYER 3 - CLASS LAYER:

This layer is specific to each implemented class. The USB specification provides the general USB communication specification, and is completely function-agnostic. Each device may implement one or more of the standard “classes” defined by the USB organization. In the case of these defined classes, the method of data transfer inside the class is also defined.

For the Mass Storage class, there are two schemes for data transfer - CBI (designed for floppy drives) and the Bulk-Only transport specification. The Bulk-Only transport is the one that is the most used, and specifies how commands and data to/from the host should be wrapped up in a well-defined manner to provide error tolerance.

LAYER 4 - SCSI Layer:

Up until now, all the layers have been completely USB-specific. However, the Mass Storage class calls for the actual medium commands (for reading, writing, status checking and the like) to be standard SCSI commands. This means that the USB transport is really just one giant wrapper for a SCSI drive - something made obvious on Linux platforms, where the kernel messages indicate a virtual SCSI driver being initiated when a USB Mass Storage device is inserted.

Once commands have been removed from the Bulk-Only wrapper, they are passed through the SCSI decoder. The SCSI layer then performs the required actions on the physical medium, resulting in data being read or written from the disk.

LAYER 5 - STORAGE LAYER:

Layers 5 and 6 are optional, and are only required if the USB device wishes to manipulate the data on the disk so that the host can read it (typical for data loggers, digital cameras and the like). This layer is able to decode the file system of the disk, so that files and folders can be added, deleted, renamed or otherwise manipulated. Because embedded devices contain limited resources, typically only one file system is supported, forcing the host to use the same file system to prevent corruption. For this reason, Mass Storage MP3 players, cameras and other devices are typically locked into using the easy to program and for and royalty free FAT file system.

LAYER 6 - FILE FORMAT:

Layer 6 is the easy one, and just entails being able to write data to the file system in the file format of choice. Loggers normally save in simple ASCII files, which require no file structure, although more complicated devices such as digital cameras need to be able to write to an image format such as JPEG.

And there you have it - a basic overview of the different layers required for a USB Mass Storage device. Hopefully this will be of some interest to embedded engineers interested in learning the USB specification.

Birthday’s are neat. While I’m not much of a consumer - at all - birthday’s do allow one to indulge slightly and make living just a little more pleasant. In the case of this birthday, I’m now a proud owner of a few more Stargate seasons, a new Maxtor Onetouch Mini 4 external hard disk for backups and a new Logitech VX Revolution wireless mouse.

Paying a little more gives a lot more benefit, I’ve come to learn. Cheap is cheap, in both quality and service. Now I’ve got a good quality Toshiba notebook and Logitech headphones and mouse, I’ve realized just how much nicer the spit-and-polish is on the name-brand products. No more Chinese cheapies for me!

One gripe so far; the Maxtor Onetouch backup software doesn’t work for me. The software detects the presence of my drive fine, but is hell-bent on indicating it as drive C:\ instead of its real drive letter allocation. Not a huge problem if I could manually set it, but as it is I can currently only backup C:\ to itself. Not a very useful proposition but one I hope to rectify with technical support.

As for the bootloader, I can now read back data correctly. Writing is still an issue which I have to tackle, but at least it’s the only remaining issue. I got back to University for year 2 in a week and a half, so I hope to have it finished completely well before then.

I thought I was being prematurely aged by my inability to get my bootloader working (not through lack of effort by any means) - turns out I’m just getting old. As of one minute ago, I’m now officially 19 years old.

Still working on getting the flash read and flash/EEPROM write commands working. I’m concentrating purely on the flash read for the moment, but it’s causing me a great deal of stress. As far as I can tell, my code’s perfect - so why I’m getting corrupted data after the first 1KB or so is beyond me. Perhaps Atmel support will be able to suggest something, but I’m not hopeful. Anyone out there want to pitch in?