Memos From the Cube

As a (hobbyist) engineer, I routinely find myself mad at Microsoft. Not because of their products (I personally find the new Office and Vista to be quite good), but because of their crud-tastic approach to engineering.

Microsoft seem happy to define their own specifications (and try to force/coerce others into adopting them), but seem totally unable to follow others. Sometimes even their own specifications are poorly implemented - like the recent OOXML debacle.

Recently I’ve been bitten by their lack of engineering abilities four times. Once, from trying to assist someone into combining Mass Storage and CDC classes into a single device. It turns out Windows won’t allow that because of the way it is built; compound devices are enumerated an interface at a time, and there’s no way to bind more than one interface in a compound device to a single OS supplied driver. At least there’s a hacky way around it, at the expense of compatibility with every other OS on the planet.

Second, through the CDC Ethernet. Admittedly there are two standards regarding this (one newer that the other, after the original was found lacking in several areas), but Microsoft went and screwed everyone over by inventing their own psudo-standard called RNDIS. Want Ethernet over USB? You’ll be writing custom drivers to get it working cross-platform.

Thirdly, their %$#@$ING STUPID XBOX Wireless design. This is a total WTF on several accounts:

1) The dongle has four fixed endpoint pairs, one for each possible paired controller. Why do that, rather than having a single endpoint pair and using some sort of index to distingush the controller to interface with?

2) The entire device and each interface is marked as a proprietary class. One would think that it’s got some super-secret, amazing protocol running underneath it to account for the custom class. Nope - one pair of the endpoints at least are just the USB HID class obfuscated to require custom drivers.

3) Despite being partly a HID device (just obfuscated so that it requires a damn custom driver), it lacks an actual HID report. It appears the returned data is supposed to be interpreted via a known, fixed layout.

They could have circumvented the whole thing by making it a compound device containing:

1) A single USB HID interface, with report descriptors indicating the button layout for the controller. Use report IDs to distinguish between different controllers. That would make the dongle enumerate on any modern OS as a bunch of standard gamepad controllers, no driver required. All the buttons, joysticks and LEDs could then be controlled via the standard HID API.

2) A single USB PID interface, to manage the force feedback. That would allow modern OSes to expose the force feedback capabilities to any modern game, no driver required.

And that’s it. No proprietary drivers, no obfuscation layers, and it would be cross-platform. Of course, any other features which are XBOX proprietary (such as memory cards, etc.) could use a legitimately proprietary additional interface in the device, but the other standard features would still be usable.

Why do they spend *extra* time to re-implement an existing standard, so that it requires more work (for them and others) to use when it has no benefit? Seems like a waste of time and money to me.

I’ve got another MyUSB API conundrum for the library users to indicate their preference about.

When I first started the project, I gave all the descriptor elements custom names which I thought were more insightful than their USB 2.0 counterparts. That was fine for me, but I’m a little worried that it might cause confusion amongst MyUSB users using the 2.0 specification for reference. Sure, the elements are in the same order as the specification and are reasonably easy to match up with the specification, but I want to put it to the users to decide what is used in the final 1.5.0 release.

There are six options that I can think of:

1) Keep everything as it is. Parameter names stay the same, as do the custom descriptor element names. Leave it up to the user (or explicitly state in the documentation) to match up the elements with their USB 2.0 Specification counterparts.

2) Replace the descriptor and function parameter names with the official USB 2.0 specification names. Would require all projects to update function prototypes and descriptors.

3) Use a union and anonymous structures (an extension of AVR-GCC) to allow both the MyUSB and the official names to be used in the descriptors (meaning that Somedescriptor.MyUSBName and Somedescriptor.officialName would both be accepted and would both refer to the same item) but keep the custom parameter names for compatibility. Would not require any changes to existing projects, but may induce confusion if part of an app uses the MyUSB names, and another uses the official names.

4) Same as the above but change the function parameter names to the USB 2.0 counterparts, forcing applications to update their function prototypes but not their descriptors. Would be the most painful option.

5) Go with option 2 if a special USB_USESPECIFICATIONNAMES or similar token is defined, otherwise default to the current MyUSB names.

6) Opposite to 5, use official names unless USB_USEOLDNAMES is defined.

Well, how about it? If I don’t get too many opinions on this I’ll just assume the majority is content with things the way they are and go with option 1.

The site’s back up on the new server, to hopefully fix the recent reliability problems — if you encounter any more problems using the new server, please drop me an email. Actually, the “new” server isn’t new at all; it’s the original server that Four Walled Cubicle was hosted on before it was moved some months ago to a (supposedly) better sever with no restrictions.

With the new server comes the new BETA of MyUSB 1.5.0, downloadable on the modified project page. I’ve changed the project page to include links to all the previous library releases (for archival purposes), as well as added downloads for the project documentation.

The site WIKI has been retired, in favor of a seperate documentation download generated from the new library source with the DoxyGen project. The new library is now fully documented, unlike the previous manual WIKI documentation, which should be a huge benefit to all the MyUSB library users. Unfortunately, that means that the documentation for older library versions are no longer accessible, however their interfaces are very close to the latest release.

The documentation is provided in HTML format, both online and as a ZIP download for convenience, and will be provided with each new release from 1.5.0 onwards. Users with DoxyGen already installed can forgo the documentation download, and instead build themselves by running a “make doxygen” in the root project folder.

MyUSB 1.5.0 is a BETA release for the moment, due to all the minor interface changes. I’ve taken this opportunity to rename some of the functions/events and remove unused API to make it more in line with the USB specification. Please send me an email if you encounter any problems with the BETA or mistakes in the documentation, however minor.

The changelog for MyUSB 1.5.0 BETA is:

• Fixed MIDI demo, now correctly waits for the endpoint to be ready between multiple note messages
• Added CDC Host demo application
• Added KeyboardFullInt demo application
• Endpoint and Pipe creation routines now mask endpoint/pipe size with the size mask, to remove transaction size bits not required for the routines (improves compatibility with devices)
• Fixed AudioInput demo - now correctly sends sampled audio to the host PC
• Shrunk round-robbin scheduler code slightly via the use of struct pointers rather than array indexes
• Fixed off-by-one error when determining if the Usage Stack is full inside the HID Report parser
• Renamed Magstripe.h to MagstripeHW.h and moved driver out of the library and into the MagStripe demo folder
• Added preprocessor checks to enable C linkage on the library components when used with a C++ compiler
• Added Still Image Host demo application
• The USB device task now restores the previously selected endpoint, allowing control requests to be transparently handled via interrupts while other endpoints are serviced through polling
• Fixed device signature being sent in reverse order in the CDC bootloader
• Host demos now have a seperate ConfigDescriptor.c/.h file for configuration descriptor processing
• HostWithParser demos now have a seperate HIDReport.c/.h file for HID report processing and dumping
• Removed non-mandatory commands from MassStorage demo to save space, fixed SENSE ResponseCode value
• CDC demos now send empty packets after sending a full one to prevent buffering issues on the host
• Updated demo descriptors to use VID/PID values donated by Atmel
• Added DoxyGen documentation to the source files
• Fixed Serial_IsCharRecieved() definition, was previously reversed
• Removed seperate USB_Descriptor_Language_t descriptor, USB_Descriptor_String_t is used instead
• Changed USB_GetDescriptor() prototype to support multiple languages
• Removed unused Device Qualifier descriptor structure
• Renamed the USB_CreateEndpoints event to the more appropriate USB_ConfigurationChanged
• Fixed MassStorageHost demo reading in the block data in reverse
• Removed outdated typedefs in StdRequestType.h, superceeded by the macro masks
• Corrected OTG.h is now included when the AVR supports both Host and Device modes, for creating OTG products
• USB_DeviceEnumerationComplete event is now also fired when in device mode and the host has finished its enumeration
• Interrupt driven demos now properly restore previously selected endpoint when ISR is complete
• USB_HOST_TIMEOUT_MS is now overridable in the user project makefile to a custom fixed timeout value
• Renamed USB_Host_SOFGeneration_* macros to more friendly USB_Host_SuspendBus(), USB_Host_ResumeBus() and USB_Host_IsBusSuspended()
• Renamed *_*_Is* macros to *_Is* to make all flag checking macros consistant, Pipe_SetInterruptFreq() is now Pipe_SetInterruptPeriod() to use the correct terminology

Thank you to all those who sent in bug reports and feature requests since the last library release. I’ve also added links to the four projects that I know of which make use of the MyUSB library on the project page — if yours is omitted leave a comment or an email and I’ll add it in.

I’m going to be uploading the new MyUSB version in the next week or so, along with the accompanying brand new DoxyGen documentation. The new documentation will replace the current site wiki documentation, which will be deleted when the new code arrives.

I’m going to modify the MyUSB project page at the same time so that previous versions of the library (including downloads and documentation from the new release onwards) will be listed, so that projects using older library releases can still be compiled without having to alter the source. These changes should move the library up in status to a real, supported library.

I’d also like to begin adding a “as used in” section to the page so that potential users of the library can see a showcase of products or projects which make use of the code. I only know of two open source projects so far to add to the list, so please email me if you’re a user!

Writing the documentation has reminded me what a tedious task it is, and I’ll be more than glad when it’s all complete. The new documentation will completely document all aspects of the public library interface, including enums, macros, functions, type defines and structures. When it is released, feel free to email me with corrections or suggestions so that it can be improved with each library iteration.

I’ve begun documenting the public interface to MyUSB using Doxygen. I’ve no experience with Doxygen before now (only JavaDoc), so it’s a new learning experience for me. I’ve managed to force it into only documenting the public sections, while correctly hiding the private sections.

This weekend the theory behind Abstract Syntax Trees - the internal grammarless representation of a computer program before it is optimized and converted into the target’s assembly code - hit me like a brick. I’ve been wondering how compilers work for a long time, but haven’t had the money nor a strong enough compulsion to purchase a book on the subject. Still, after working with grammar trees in Discrete Mathmatics at Univeristy last week, the idea struck me without further prompting.

I’ve submitted a tentative patch to the dfu-programmer project to add in EEPROM reading and writing support, as I found that it lacked EEPROM functionality. That will hopefully be applied soon, allowing non-windows users to use the MyUSB DFU bootloader, or even Atmel’s DFU bootloader.

Thanks for the pointers to the documentation projects. I’m still tossing up what to use; I don’t want to clutter the source up, and I don’t want some of the exact implementation details to be shown to the user (for example, I’d want the endpoint macros to appear as functions instead).

I spent last night tweaking the Mass Storage device demo to make it smaller and faster. By downloading the Mass Storage Compliance document from the USB-IF I was able to rip out all the SCSI commands not made mandatory by the specification. The result is sizable reduction in code, saving space, potential bugs and comprehension requirements. The demo currently writes at a peak of a little over 100KB/S on Vista 32-bit, and reads at over 200KB/S on the same system, and compiles down to a little over 5KB.

Speaking of bugs, the code cleanup revealed an annoying bug in the Mass Storage code. After re-testing the demo with an Ubuntu live CD, I discovered that the demo wouldn’t mount with the latest code. Further probing caused me to bang my head on the wall in shame; the SCSI SENSE data being returned by my code had an invalid parameter which Windows ignored, but Ubuntu (specifically, the Linux SCSI handler) chucked a wobbly over.

Looks like we’re back in business after a second period of downtime. I was pleasantly surprised to find people actively seeking out alternative download locations for my projects — while the server downtime was unfortunate, I’m motivated knowing people are interested in what I’m working on.

I’ve completed an initial implementation of my Still Image Host demo, for the next library release. Unfortunately my camera doesn’t seem to implement many interesting PIMA commands, so instead I’ve left much of the functionality up to the end-user, with the demo only displaying data extracted from the device info dataset, and opening/closing sessions with the attached camera. The API for sending and receiving PIMA commands (the standard for remote camera manipulation) in the demo is defined, so it should be simple to issue and process any additional desired commands.

I’ve been musing over adding in some automated documentation to the project, via a project like DoxyGen or similar. The advantage would be automatic, always correct documentation - a huge plus. The disadvantage would be all the massive comment blocks needed for every library component (define, typedef, etc.). Right now I think an alternative to a manual wiki is needed, but can’t think of an appropriate solution, so feel free to suggest one.

My apologies for the downtime yesterday. Haven’t a clue what caused it — looks like a problem with the server provider. Whatever the reason, everything is once again working as it should.

I’ve given my blog a little bit of a face lift via a new WordPress theme. I actually stumbled upon it while looking for a theme for the comical “Glacial Pace” blog my University friends are planning on starting up (based around my friend’s notoriously slow eating pace). While I can’t be certain, the new theme carries distinct similarities to the old, so it probably a newer version by the theme author. In any case, it spruces things up slightly without radically altering the design of the blog so I’ve switched to it.

Based on the few comments to my last entry plus research of my own, I’m thinking of leaving the Ethernet-Over-USB project for a little while. I’ve re-opened my unreleased “StillImageHost” demo code, which will eventually allow PIMA-compatible USB cameras (that’s anything which enumerates as a camera device rather than a Mass Storage device under Windows) to be controlled via the USB AVRs. I’ve already got a few of the commands working, so expect more progress on that soon.

Leave it to Microsoft to shoot down neat project ideas. I decided to get stuck into the Ethernet (and the HTTP and TCP/IP) protocols so I could make an embedded webserver using the USBKEY, with the Ethernet data being transmitted to the computer over USB using the Ethernet subclass of the CDC class.

Well, as usual, Microsoft doesn’t want to play ball. I’ve been trying to make all my MyUSB demos platform agnostic, but I’m in a bit of a bind. Microsoft haven’t implemented the Ethernet subclass of the CDC standard class, nor have they implemented the newer EEM CDC subclass. Instead they’ve gone and developed their own “RNDIS” subclass which contains proprietary extentions.
That makes perfect sense. Why implement a standard defined class, so defined as to promote platform and driver independence? Why not go for a slightly different method instead, breaking compatibility with the existing defined subclass that everyone else supports?

That puts me into a bind. I have several options:

1) Give Windows the finger, and develop for the standard EEM or Ethernet subclass. Require someone to write a EEM or Ethernet CDC subclass driver to make the demo compatible with Windows (note, these exist, but the rubbish “Jungo” company wants $4000 for a developer’s licence).

2) Ignore the standard and all other OSes, and develop a demo for Microsoft’s proprietary standard. Make the demo compatible with Windows, but not work on any other OS (unless the Linux developers have made a RNDIS driver).

3) Forget it, and concentrate on other things. Get back to writing my book, or developing other demos/parts of the MyUSB library.

4) Develop two demos with identical functionality, but using the different standards. This one doesn’t really appeal to me.

5) Make one demo, with dual standards - either via a compile time option, or some other method. I’d just expose both standards via the descriptor and let the OS sort it out, but Windows has a habit of not loading its native drivers for any compliant device that also contains non-standard interfaces.

I guess I’ll put it to public vote. Which option should I take?

EDIT: Some people are wondering about the utility of an embedded Ethernet webserver running over USB. I see two reasons to write it:

1) Because I can, and because it would serve as a demo for the CDC EEM class standard.

2) Because it would provide a more modern configuration interface to a complicated device. Think of it as a replacement for configuration via serial terminals - this way the user can configure the device over USB with nothing more than a standard web browser.

Today I added the twenty-second demo to the MyUSB project; a CDC host demo. It’s very basic: enumerated CDC devices using the ACM profile have their sent data relayed through the serial port. Think a reverse USB-Serial converter, where an actual USB-Serial converter device plugs into the USB AVR (forming a RS-232 to logic level serial converter, using USB as the transport). I haven’t been able to properly test it however, as all my converters report non-standard class/subclass/protocol values in their descriptors to force the use of their vendor-specific drivers. That’s progress for you; create a defined class to allow different vendors to supply devices which can be driven from a common driver, and they’ll find a way to cock up the implementation.

Speaking of mistakes, I also fixed the AudioInput demo for the next version. The supplied code can’t possibly work (endpoint is created using the wrong direction token, the sample timer is continuously reset because I forgot to set the HasConfiguredTimer flag), yet I remember testing it; I suspect I just had the PC set to the wrong input source and was actually seeing data from my laptop’s internal microphone. Go figure. The updated code will be in the next MyUSB version release, a while from now. As always, if you need the updated code right this minute, send me an email.

EDIT: Neat, my 127th post, the maximum positive value of a signed 8-bit integer. Although I always thought I’d have at least one regular reader by the 100th post…