Memos From the Cube

Good news!
After sloughing through my own TCP/IP stack for a week now, I’ve finally got some usable results. I’m now able to connect to the device’s simple shell through TELNET, and see a basic webpage when directing to the fixed IP address. It’s got a lot of problems to fix yet (haven’t added the code to create multiple TCP transactions when a large amount of data is sent at once, the webserver doesn’t terminate the connection) but it works damnit!

The code is somewhat simplified by the lack of a proper ARP system (at the moment). Because the USB AVR enumerates to the host as an actual, seperate network adapter, implementing a full ARP table would be pointless, as only the host computer is able to communicate with it. Still, despite the lack of a proper ARP, the code allows for multiple ports to be opened, and applications to consume and create TCP data as needed.

I have to say I’m quite proud of it, despite the god-awful code. I’m going to try to clean it up a bit first, then complete it and put it as a demo in the next MyUSB revision.

Good news! About a week ago, I managed to get an initial implementation of Microsoft’s RNDIS specification working as a MyUSB demo, so that the USB AVRs could be enumerated as a network adapter on modern OSes (yes, Linux has RNDIS support now).

That implementation made it into the 1.5.1 MyUSB release two days ago. However, the demo was limited; it could only send/receive raw Ethernet frames, it didn’t perform any processing on them as that was left up to the application engineer. It also failed to work on older Linux kernels with the original RNDIS implementation, due to a flaw in the Linux code — the notification endpoint was ignored in favour of polling the control endpoint, causing a lockup while the device waited for the notification endpoint to become empty again.

Both of those issues have now been resolved in the latest working code, with a big thanks to AVRFreaks member Colin O’Flynn for his advice and debugging help. The demo now works on the older RNDIS Linux code, and it can do something with the Ethernet frames!
Currently, I’ve written up protocol handlers for Ethernet, ARP, IP and ICMP, with TCP a work in progress. That’s the bare minimum to get a pingable device:

Which is damn cool in my opinion — if only because I’ve spent a whole week getting this far. Once I get TCP working, I’ll be able to write up a simple webserver, with the ultimate goal of being able to control the board LEDs using a web browser.

What’s the utility of all this? Think how much easier a device would be to configure if you could just plug it into a laptop, navigate to it with a web browser (on any platform) and change configuration settings using native widgets. No more complex serial terminals!

Stay tuned for updates.

Additional: For those with good eyesight and inquisitive minds - the 300ms+ ping time is due to the USBKEY printing out the protocol details of each packet through the serial port for debugging.

I’ve decided to release MyUSB 1.5.1 today. This new version is a very minor update, mainly fixing errors in the demos — in fact, there are no migration issues at all from the previous 1.5.0 version. Other small enhancements to the library itself (including one or two fixes to long standing bugs) are present in the new version, but these are invisible other than flash code size savings.

Without further ado, the changelog for the new version:

• Changed host demos to enable the host function task on the firing of the USB_DeviceEnumerationComplete event rather than the USB_DeviceAttached event

• HID Usage Stack now forcefully cleared after an IN/OUT/FEATURE item has been completely processed to remove any referenced but not created usages

• Changed USB_INT_DisableAllInterrupts() and USB_INT_ClearAllInterrupts(), USB_Host_GetNextDescriptorOfType(), USB_Host_GetNextDescriptorOfTypeBefore(), USB_Host_GetNextDescriptorOfTypeAfter() to normal functions (from inline)

• Fixed USBtoSerial demo not sending data, only receiving

• Fixed main makefile to make all by default, fixed MagStripe directory case to prevent case-sensitive path problems

• ConfigDescriptor functions made normal, instead of static inline

• Pipe/Endpoint *_Ignore_* functions changed to *_Discard_*, old names still present as aliases

• Fixed ENDPOINT_MAX_SIZE define to be correct on limited USB controller AVRs

• Changed endpoint and pipe size translation routines to use previous IF/ELSE IF cascade code, new algorithmic approach was buggy and caused problems

• Bootloaders now compile with -fno-inline-small-functions option to reduce code size

• Audio demos now use correct endpoint sizes for full and limited controller USB AVRs, double banking in all cases to be in line with the specification (isochronous endpoints MUST be double banked)

• Added Interface Association descriptor to StdDescriptors.h, based on the relevant USB2.0 ECN

• Fixed MIDI demo, corrected Audio Streaming descriptor to follow the MIDI-specific AS structure

• Fixed HID class demo descriptors so that the HID interface’s protocol is 0×00 (required for non-boot protocol HID devices) to prevent problems on hosts expecting the boot protocol functions to be supported

• Added read/write control stream functions to Endpoint.h

• Fixed AudioOut demo not setting port pins to inputs on USB disconnect properly

• Added RNDISEthernet demo application

As always, email me if you have any suggestions or bug reports. Download the new library code from the project page. Also on that page is the latest library documentation, in downloadable or online HTML form.

A few days work, and what do I have to show for it? Only this:

Why is it showing as unplugged? I’ve no idea — I’m going to test it out on Ubuntu and use the dmesg tool to see if I can discover what’s going on. Definite progress has been made however, as there’s quite a bit going on behind the scenes. I can’t wait to get all the RNDIS wrapping completed so I can then start attacking the Ethernet frames. My goal for the moment is a pingable device (hard enough in itself) but eventually I want to have a micro HTTP server running on this thing. Lots of layers between me and that goal - Ethernet, TCP, IP, HTTP - but by taking it one step at a time it should be perfectly possible for myself to manage.

To some more good news; I’m very interested in the creations over at www.digitalsurveyinstruments.com, not because I want to become a geologist or surveyer, but because the devices are running - you guessed it! - MyUSB. I’ve had a ball talking to the engineer behind the products, discussing various ways to acomplish certain tasks, and ranting when it turns out that Windows doesn’t support what we want.

UPDATE: Found the source of the “Network Cable Unplugged” problem: RNDIS_STATUS_MEDIA_CONNECT isn’t the same as RNDIS_MEDIA_STATE_CONNECTED. It’s Ethernet time, baby!

Today I received a rather nice email from fellow AVRFreaks member Collin O’Flynn. In part:

Hi Dean,

I ended up getting RNDIS support to work on the Atmel USB chipset.

I used the Atmel application note as a base, due to the fact that you need a
lot of other crap for this to be useful. Other crap means TCP/IP stack of
course… and the Atmel application note was a bit easier to integrate
beacuse of this.

But I’m trying to get the code to interface to Linux now, as Linux does have
RNDIS support and should have working IAD support….

That’s rather neat - it means that a working RNDIS implementation is possible on the USB AVRs. I’ve been ITCHING for something to do the last two days, so I’m jumping in and cooking up my own implementation. Everything is from scratch — I’ve re-downloaded the RNDIS specification once again and started coding, basing the demo on the existing CDC MyUSB demo. Hopefully I’ll be able to make a working implementation on top of MyUSB without the use of any RTOS or dynamic memory.

I know I originally discounted the idea of Ethernet over USB due to the three competing standards (two of which are USB-IF certified, and once of which - RNDIS - is a Microsoft abomination) however since it appears that Linux supports it I can’t see the harm. In any case, it’s a sad fact that the majority of systems run Windows. And let’s face it, it’s many magnitudes more likely that Linux will implement RNDIS (quckly!) than it is that Microsoft will implement one of the true CDC Ethernet standards.

So far I’ve got my implementation to enumerate and show up in the Network Connections screen of my laptop, although the interface is currently dead (disabled permenantly) until I implement all the mandatory control commands and start sending some data.

I’m also excited to learn that the first 1000 board run of Donald Davis’ “Benito” Arduino programmer boards is going ahead. Excited because it’s the first large scale production of a device (to my knowledge) running MyUSB! If your project runs on MyUSB, please don’t forget to send me an email!

Right, soon 1.5.1 will be released - found a few more bugs which needed to be quashed, especially the faulty endpoint/pipe size to mask translation routine. One small new thing that will make it into the revision release will be the addition of the Interface Association standard descriptor to StdDescriptors.h.

The IAD is something I stumbled onto by chance while trying to find a link to a thread discussing getting CDC + anything to run as a compound device under Windows. In Windows, devices implementing several classes - called Compound devices - have each interface enumerated separately will different drivers. For most devices that isn’t a problem, but in the case of the CDC class it is disasterous, as the CDC class uses two separate interfaces for the one function.

Having a basic CDC class device is fine under Windows, as Windows then passes the entire device over to the usbser.sys driver file and all is well. However, under compound device mode, the two CDC interfaces are treated as separate functions and the result is a non-functioning CDC. People have posted Windows-specific workarounds based on the shoddyness of the drivers to get it to work, however that breaks functionality on other OSes.

While trying to find the thread which explained all that for the person enquiring about it, I found a mention of the “Interface Association Descriptor”, something I’d missed until now. It turns out that the IAD was a supplement to the USB2.0 standard and was posted as an Engineering Change Notice (ECN) after the specification, meaning that unless I knew to look for it, I wouldn’t have found it.

The IAD is designed to address exactly the above problem. It allows a compound device to specify which interfaces should be grouped and enumerated together under the one driver - allowing the two CDC interfaces to be linked, for example. Damn useful, with only two drawbacks:

1) It is only supported in Windows XP SP2 or higher, and in newer Linux releases
2) It requires the device to use a special class indicating that it is a IAD supporting device, meaning non supporting OSes will fail to enumerate the device completely rather than just some of the functions

However, it will be very useful in the future for all new devices, especially those for Vista and beyond. I’m really glad to have a real solution to the problem, so I’m adding in the descriptor to MyUSB.

You can read the IAD document here.

In case anyone is wondering, now is mid-year University exam time, which is the main reason for my lack of updates here. Actually, truth be told, discovering the (modern) Battlestar Galactica series has been the main contributor towards my laziness.

After tomorrow I’ll be free from Univeristy for a whole month. That’s great, but I’m going into my usual axiety spiral; not knowing what exactly to do any one one moment, causing me to do nothing and stress about wasting my free time. If my new contract job eventually gets of the ground I’ll have both a source of income (finally - ramen noodles, sleeping on burlap sacks and begging are tiresome!) and a sink for my creative *cough* tallent.

Soon I think I’ll release MyUSB 1.5.1, a very minor update fixing a couple of bugs in the demos and other non-critical aspects of the library like typos and the like. The library is for all intents and purposes done, so barring an ephiphany all updates in the near future will be minor only - no more frenzied development of new APIs.

It’s probably about time I got back to writing my USB book, which I’ve neglected. Hang on, I think I hear the next Battlestar Galactica episode calling me…

Well unfortunately today my laptop’s Vista install decided that six months was far too long living and committed suicide. Eye-witness reports indicate that a bungled auto-update was leaving the scene shortly before the explosion which took out the OS foundations.

I managed to fix the installation with a quick OS repair, but ended up deciding to copy off my data, wipe the disk and start again. The first month or so of owning the laptop was a hard time for the OS, as my Toshiba laptop model’s “black screen of death” (since fixed with a BIOS update) caused many unscheduled system crashes which no doubt would have bamboozled some of the critical components in unknowable ways. That, and the crashes from my early USB experiments led to me decide to reinstall all over again.

Several hours later, and it’s like I never left - score one for external hard disks. I’m still fuming about the latest trend of using hidden “recovery partitions” rather than actual physical OS install disks, but to Toshiba’s credit the re-imaging of the disk did go along smoothly. The only casualty which I’ll be morning for many months yet was an obscure old game called Lands Of Lore which I had installed into a part of my computer outside my profile, causing me to loose the saved game files. That really sucks, as I’ve had the game for at least 10 years and my brother and I are yet to finish it, and the saved game was at the very last obstacle before the end of the game. Oh, how I lament my forgetfulness!

Oh well, at least everything else is back to normal and stable once again. Now for the 9 trillion OS updates…

MyUSB 1.5.0 has been released onto the project page, including the new documentation. I completed the migration information I posted about a few hours ago - it is now a page in the online and downloadable documentation. Please notify me of any problems found with the new version.
Previous 1.5.0 BETAs have been removed in the wake of the new release and are no longer available for download. The full changelog for the new version is:

• Fixed MIDI demo, now correctly waits for the endpoint to be ready between multiple note messages
• Added CDC Host demo application
• Added KeyboardFullInt demo application
• Endpoint and Pipe creation routines now mask endpoint/pipe size with the size mask, to remove transaction size bits not required for the routines (improves compatibility with devices)
• Fixed AudioInput demo - now correctly sends sampled audio to the host PC
• Fixed AudioOutput demo oncemore — apparently Windows requires endpoint packets to be >=192 bytes
• Shrunk round-robbin scheduler code slightly via the use of struct pointers rather than array indexes
• Fixed off-by-one error when determining if the Usage Stack is full inside the HID Report parser
• Renamed Magstripe.h to MagstripeHW.h and moved driver out of the library and into the MagStripe demo folder
• Added preprocessor checks to enable C linkage on the library components when used with a C++ compiler
• Added Still Image Host demo application
• The USB device task now restores the previously selected endpoint, allowing control requests to be transparently handled via interrupts while other endpoints are serviced through polling
• Fixed device signature being sent in reverse order in the CDC bootloader
• Host demos now have a seperate ConfigDescriptor.c/.h file for configuration descriptor processing
• HostWithParser demos now have a seperate HIDReport.c/.h file for HID report processing and dumping
• Removed non-mandatory commands from MassStorage demo to save space, fixed SENSE ResponseCode value
• CDC demos now send empty packets after sending a full one to prevent buffering issues on the host
• Updated demo descriptors to use VID/PID values donated by Atmel
• Added DoxyGen documentation to the source files
• Fixed Serial_IsCharRecieved() definition, was previously reversed
• Removed seperate USB_Descriptor_Language_t descriptor, USB_Descriptor_String_t is used instead
• Removed unused Device Qualifier descriptor structure
• Renamed the USB_CreateEndpoints event to the more appropriate USB_ConfigurationChanged
• Fixed MassStorageHost demo reading in the block data in reverse
• Removed outdated typedefs in StdRequestType.h, superceeded by the macro masks
• Corrected OTG.h is now included when the AVR supports both Host and Device modes, for creating OTG products
• USB_DeviceEnumerationComplete event is now also fired when in device mode and the host has finished its enumeration
• Interrupt driven demos now properly restore previously selected endpoint when ISR is complete
• USB_HOST_TIMEOUT_MS is now overridable in the user project makefile to a custom fixed timeout value
• Renamed USB_Host_SOFGeneration_* macros to more friendly USB_Host_SuspendBus(), USB_Host_ResumeBus() and USB_Host_IsBusSuspended()
• Renamed *_*_Is* macros to *_Is* to make all flag checking macros consistant, Pipe_SetInterruptFreq() is now Pipe_SetInterruptPeriod() to use the correct terminology
• UnicodeString member of USB_Descriptor_String_t struct changed to an ordinary int array type, so that the GCC Unicode strings (prefixed with an L before the opening quotation mark) can be used instead of explicit arrays of ASCII characters
• Fixed Endpoint/Pipes being configured incorrectly if the maximum endpoint/pipe size for the selected USB AVR model was given as the bank size
• HID device demos now use a true raw array for the HID report descriptor rather than a struct wrapped array
• Added VERSION_BCD() macro, fixed reported HID and USB version numbers in demo descriptors
• Cleaned up GetDescriptor device chapter 9 handler function
• Added GET_REPORT class specific request to HID demos to make them complaint to the HID class
• Cleaned up setting of USB_IsInitialized and USB_IsConnected values to only when needed
• Removed Atomic.c and ISRMacro.h; the library was already only compatible with recent avr-lib-c for other reasons
• All demos and library functions now use USB standardized names for the USB data (bRequest, wLength, etc.)
• Added USE_NONSTANDARD_DESCRIPTOR_NAMES token to switch back to the non-standard descriptor element names

Turns out I’ve gone and made a liar out of myself, by making more API changes in preparation for the 1.5.0 MyUSB release. The final changes have been made to the descriptor and request APIs.

As I indicated previously, I’ve changed the descriptor structures to use the official names by default. The previous names can still be used however (in fact, I still prefer them in the case of the descriptors, thus the demos use the non-standard names) by defining the token USE_NONSTANDARD_DESCRIPTOR_NAMES in the project makefile. That should satisfy both camps, as the programmer now gets a choice over which scheme to use. Of course, the library internals work correctly with either option.

In a similar vein, the USB_UnhandledControlPacket event now uses the official names for the request parameters - bRequest and bmRequestType instead of Request and RequestType. That small change is reflected in the library internals also, as all request variables now use the official names rather than my own. Using the official names for the parameters reduces confusion amongst developers, as now the descriptor and parameter names match up to the official specification with no room for misinterpretation.

One of the pressing reasons for the change to the official names was the GetDescriptor function, which now has parameters for the full wValue and wIndex values rather than separated and renamed values. Many USB class specifications use the GetDescriptor request to return class-specific data, and in these instances the parameter values take on whole new meanings, which the non-standard names did not reflect adequately.

Host projects will also need to update, as they will be using the non-standard names in the configuration descriptor processing. Like with the device projects, the old names can be brought back with the defining of the new USE_NONSTANDARD_DESCRIPTOR_NAMES token.

A cavet; while the descriptor names can be switched back to the non-standard names, the parameters and the host mode request structure element names cannot. That is because I think that the non-standard descriptor names can add value, while the other non-standard names only add confusion.