Memos From the Cube

My mid-year exams completed, I am now officially in holiday mode! I’ll be leaving tomorrow with family (and my girlfriend, lest she read this when she gets back and feels left out) for a short cruise to Numea until the 10th, and will be out of contact. Sorry in advance to those needing to get a hold of me during that time. I will most likely have no internet access at all until my return, so I ask people with questions for me to be patient.

With any luck, the world won’t implode in my absence and I’ll be able to return relaxed and ready to jump back in to LUFA development and the other 9,000 things I need to do.

Huzzah!

Well, so far I’ve received only a few logos, but I’ll post them all here for public scrutiny. The next LUFA release will not be until at least mid-to-late June so there’s still plenty of time to figure out what will take the place of the official LUFA project logo. Without further ado, here’s the possibilities so far:

The adge-old “Beggar’s can’t be choosers” doesn’t totally apply here – I begged, and we now get to choose from the offerings provided. There’s still time if you wish to submit your own design, but in the event of no more entries we need to chose one of the above to serve as the masthead for the project. Comment on your favourite, and be kind.

I will be away on a holiday from the 2/07/09 to the 10/07/09 and will be most likely be 100% unreachable – so if you have any burning queries, get them in now lest you have to wait until I get back. Although, to be honest, if your query is burning, I suggest you see a doctor or call the fire bridage as appropriate.

I think, after all this time, that LUFA needs a project logo. Everything these days has a logo of some sort for branding, and I think LUFA should be no exception even if it is a free product. As such, I’m putting the call out for anyone handy with a graphics editor (i.e. not me!) to come up with possible logos. I’ll put up all the designs I receive so that I can get a public vote on the best one, which will be used in the manual, project page and public SVN.

So far, I’ve already received one set of designs from Pavla Dlab, shown below – if you have an opinion on these, comment! If you have a design you wish to submit, please email it to me directly at dean {at} fourwalledcubicle [dot] com.

Apart from just being damn cool, the following is also being partly driven by LUFA:

Totally awesome.

All’s been unsettlingly quiet this last week; almost no discussion around the latest release at all. Usually I have many LUFA support requests (both on the mailing list, and via private email) banked up with no time to answer them all, so I’m a little disconcerted to actually have an empty inbox. For now, I’m going to assume that everyone is happy with the new release and no one’s having any problems — although the truth is probably more along the lines of no one’s bothered to upgrade.

This month is exams month; I am busy studying – well, partially studying – for my midyear exams which start at the end of this week. For this reason I’m not quite as active as I usually am in developing and replying to messages, but I hope everyone will just bare with me until the end of the month when I can go back to being a crazed caffeine-fuelled coding maniac. Early next month I’ll be sailing in a “small bathtub” (the Pacific Dawn, a mega-cruse ship which has featured prominently on the national Australian news for its Swine-infected crew these last few weeks) with my family for a week.

I’ve almost completed the device-mode class drivers for LUFA, although they are still a work-in-progress as I figure out how to design the corresponding host mode drivers. My goal for the host mode drivers are to allow for multiple drivers to be used in the one app, so that several classes of attached devices can be driven from the one one application easily. That’s not the same thing as driving multiple devices at once; this will only allow for a single device, but will allow for different class interfaces to be driven from the different drivers in the one application. That’s causing me a little grief working out how it will all fit together, but the results will be worth it.

I was a little astonished to go over the LUFA codebase earlier today. From such humble beginnings, the exported repository now weighs over 6MB, contains well over 500 source files and supports all the released Atmel USB AVRs and USB AVR boards. I realise that many of those files are duplicates thanks to the new Class and Low Level demos, but I still can’t believe that I’ve managed to accomplish so much.

As the current release is to be the “stable” API, I see no reason currently to continue with monthly releases at the expense of quality. Assuming no critical bugs are found, I’m comfortable in light of my exams and holiday to push back next month’s release to the end of the month, to ensure that all the new features are as complete and high quality as possible. The planned featureset is as follows:

1. Removal of the cruddy “Scheduler” from all demos, switch to true loops instead for clarity (complete)
2. Better library documentation page structure, with sub-page grouping (complete)
3. New Device mode class drivers for easy implementation of the standard USB classes (complete)
4. Documentation of the new Device mode class drivers (complete)
5. Creation of new demos using the Device mode class drivers to be included alongside existing low level demos (complete)
6. New Host mode class drivers for easy implementation of the standard USB classes as hosts (in progress)
7. Documentation of the new Host mode class drivers
8. Creation of new demos using the Host mode class drivers to be included alongside existing low level demos

So as one can see, the focus is on the new class driver framework which can be used in preference to the low level APIs for rapid application design. This means duplicating all the existing demos and re-implementing them using the class driver framework to give the same results using the different API layer.

On a final note to this rather eclectic post; I received an interesting comment from “Truth”, stating that all the compatibility troubles we’ve been experiencing with the Mass Storage Host demo can be tracked down to the routine waiting for data to be received, where it unfreezes both data pipes at once. It seems not all USB controllers can handle this, and it causes them to lock up. I’ve patched the function in the SVN to only unfreeze one pipe at a time, and it seems to have helped on the sticks I’ve tested, with all four working flawlessly. Hopefully that will mean an end to compatibility problems with USB flash drives from the next release.

Late last week a new package arrived on my doorstep, containing the new product lineup from AVROpendous. Matt has always been very generous with sending me samples of his new hardware, and I can’t say no to packages (although, on occasion, they so no to me and end up flying around Belgium for not apparent reason).

The three big contenders in the consumer USB AVR market (making USB AVR based boards for hobbyists and experimenters) are AVROpendous with the Opendous boards, PJRC with the Teensy boards and of course Atmel with their official USB AVR evaluation boards. This is a great step forward, since a year ago two of those didn’t exist, and USB AVRs were the realm of commercial enterprises with lots of time and money on their hands.

Of the two smaller companies, AVROpendous and PJRC have been engaged in somewhat of an innovation war, trying to grab customers from the same limited market. As such they have taken slightly different directions; Matt, with the open source/open hardware AVROpendous, appeals to the open source crowd. On the flipside, Paul over at PJRC has been trying to win over the novice crowd with the Teensy and its “HalfKay” bootloader, has a closed design but open source (except for the bootloader).

From the beginning, the designs of the two were very different: AVROpendous V1 had an innovative SIL layout, inserting vertically into a breadboard for maximum space savings, while the Teensy had a DIP layout similar to the old – and entirely useless – BASIC Stamp. Since then AVROpendous has also brought out the AVROpendous Mini, a smaller board which used female sockets instead of pin headers and so was wired to a breadboard rather than inserted into it.

Now it’s time for the next wave of designs. PJRC has brought out the Teensy++ sporting a larger ATMEGA32U4 AT90USB646 microcontroller, while AVROpendous has gone for a similar DIP design but with four different versions.

First up, is the tiny AVROpendous-DIP, essentially a clone of the original Teensy – although it has seperate buttons for HWB and RESET rather than the Teensy’s single all-in-one button. I liked the combined button of the Teensy for its super-ease-of-use while programming but found it limiting when I just wanted to reset the AVR and not reprogram it, or just have a “Do Something” hardware button prewired onto the board. This version of the AVROpendous uses the same AT90USB162 microcontroller used on all previous AVROpendous boards, plus the original Teensy.

Second is the interesting AVROpendous2-DIP, which is a slightly larger version of the board mentioned above, due to its use of the newer ATMEGA32U4  microcontroller. This is my first ATMEGA32U4 board and so I’ll be happy to finally have a test-target for the U4 series USB AVRs. Due to space constraints Matt’s used the smaller pin header layout for the JTAG which no one can easily use — unless you got the adapter in the Atmel Raven kits, that is. Got to love free hardware!

Thirdly is one board Matt didn’t send me, the AVROpendous-3D, because it is a subset of the largest board. Sporting a AT90USB646 microcontroller, it provides plenty of FLASH space, a normal JTAG layout and device-only USB operation for the development of complex USB peripherals. This is the same AVR model used on the new Teensy++ board.

Finally is the last board in Matt’s package to me, the AVROpendous-3H. This is the superset of the 3D above and contains a AT90USB647 microcontroller, and so provides USB Hosting capabilities. This board is unusual for two reasons; first, it’s the first non-Atmel general USB AVR board to offer a USB AVR that supports USB hosting mode, and secondly, it contains two USB connectors. The first connector is the standard mini-USB B connector on all the other Opendous/Teensy/USBKEY/etc. boards, but the second is a full-size USB-A connector mounted right on the board. Rather than have the user work out where to get a mini-USB a to USB-A converter cable, Matt decided to just put a second connector on the board for USB hosting and have a jumper select between the two. It’s an unusual move, but after using it for a while I have to say it’s more convenient than the USBKEY, as it means that the mini-USB B connector can be left into the host for reprogramming and powering the board.

All in all, an impressive lineup. Matt’s busy working on porting his AVROpendous package to the latest LUFA release, so users of his boards stay tuned! As mentioned, this month’s release holds the last of the major API changes, with future releases focusing on preserving backwards compatibility. If you haven’t already upgraded, get to it already!

I’ve gone ahead and released the next LUFA revision to the unwashed masses – get it here. This is based off the current public SVN revision and thus doesn’t have the new class-level driver APIs I’m currently working on, but I felt that it was worth pushing that forward to next month so I can get this release out the door. This release aims to simplfy and finalise the low-level APIs, serving as a stable base for future revisions. The only real difference between this release and the next as far as a the low-level APIs go will be that the pseudo scheduler (which wasn’t really a RTOS at all and caused all sorts of confusion) will be permenantly removed.

Most of the migration notes for this release center around remvoving things rather than adding – most of the abstraction macros have gone. Events now use regular function syntax and thus have easy-to-see parameters, for example, leading to clearer code. I really don’t expect much grumbling from users on this version, and none at all from the next as the low level APIs will no longer change. Everything is finally as it should be .

As always, the changelog is reproduced below:

• Fixed bug in RNDISEthernet and DualCDC demos not using the correct USB_ControlRequest structure for control request data
• Fixed documentation showing incorrect USB mode support on the supported AVRs list
• Fixed RNDISEthernet not working under Linux due to Linux requiring an “optional” RNDIS request which was unhandled
• Fixed Mouse and Keyboard device demos not acting in accordance with the HID specification for idle periods (thanks to Brian Dickman)
• Removed support for endpoint/pipe non-control interrupts; these did not act in the way users expected, and had many subtle issues
• Fixed Device Mode not handling Set Feature and Clear Feature Chapter 9 requests that are addressed to the device (thanks to Brian Dickman)
• Moved control endpoint interrupt handling into the library itself, enable via the new INTERRUPT_CONTROL_ENDPOINT token
• Fixed CDCHost not clearing configured pipes and resetting configured pipes mask when a partially enumerated invalid CDC interface is skipped
• Clarified the size of library tokens which accept integer values in the Compile Time Tokens page, values now use the smallest datatype inside the library that is able to hold their defined value to save space
• Removed DESCRIPTOR_ADDRESS() macro as it was largely supurflous and only served to obfuscate code
• Rewritten event system to remove all macros, to make user code clearer
• Fixed incorrect ENDPOINT_EPNUM_MASK mask preventing endpoints above EP3 from being selected (thanks to Jonathan Oakley)
• Removed STREAM_CALLBACK() macro – callbacks now use regular function definitions to clarify user code
• Removed DESCRIPTOR_COMPARATOR() macro – comparators should now use regular function definitions to clarify user code
• USB_IsConnected is now cleared before the USB_Disconnect() event is fired in response to VBUS being removed
• Fixed incorrect PID value being used in the USBtoSerial project (thanks to Phill)
• Deleted StdDescriptors.c, renamed USB_GetDescriptor() to CALLBACK_USB_GetDescriptor, moved ConfigDescriptor.c/.h from the LUFA/Drivers/USB/Class/ directory to LUFA/Drivers/USB/HighLevel/ in preperation for the new USB class APIs
• Moved out each demos’ functionality library files (e.g. Ring Buffer library) to /Lib directories for a better directory structure
• Removed Tx interrupt from the USBtoSerial demo; now sends characters via polling to ensure more time for the Rx interrupt

As you can see, most of it is actually bug fixes rather than new additions, another sign that stability is finally comming to the project.

Those who wish to check out the upcomming class driver changes can do so – see this thread for details.

Wow. I was not expecting the level of discussion (hatred?) I have seen about the new LUFA APIs. Working on LUFA has given me much insight to the workings of real-world projects — it’s forced me to learn better source revision techniques, and the ins-and-outs of managing a project used by many different people worldwide.

One thing I’ve taken from the recent abuse correspondance about the library is that users hate change. Specifically, users hate things changing between versions. Having stood back, I fully agree; the LUFA codebase has changed way too much between versions, since I’ve been subscribing to a programming methodology best suited to a private project (make changes for performance/efficiency always, even if it breaks APIs).  This is a poor stance for a library which is meant to be used by others, so that will stop in the next release.

Let me reiterate; with the next LUFA release, I will endevour to keep all the APIs the same, unless there is a pressing need for a breaking change. One such change came recently when I was informed that the endpoint/pipe clearing APIs weren’t suited to the edge-case of received empty packets from the host, which caused me to have to completely alter them to suit. Other changes have been made recently for the sake of simplicity and consistency, much to the consternation of users.

So, here’s the deal – take it or leave it. The next LUFA version will carry big changes, for the sake of simplifying things. This will require more changes in user code, as has been the case previously. However, upgrades beyond the next release should be a painless drop-in replacement affair, unless clearly documented. All the demos will use the new class driver layers I’m working on, to simplify them and reduce the amount of support required for the library, the cruddy not-quite-a-scheduler scheduler will be removed (although users can copy it from an old version into their project if they wish) and the event system/library named callbacks have changed and will require minor user tweaking. Once that’s done, we can all look forward to painless updates and I can look forward to a lot less negative discussion about LUFA.

One more thing that bears reiteration to make certain it’s clear; the new class drivers are a layer on top of the low level APIs for convenience; they do not replace the low level APIs. If users wish to retain their low level code (which is neccesary if they are implementing classes not covered in the library) they can do so.

I’ve come to realise that the current design of the LUFA API was a bad idea in many respects. Most people were confused by the simple “Scheduler” included in the demos, which was just a nice way (IMHO, of course) of thinking about the multitude of functions normally called in a loop within a project’s main function. Many people took a look at all the class boilerplate code required to implement each USB class in the demos and spat the dummy – I’m looking at you, Arduino programmers – and were turned off using the library.

As such, now that I’ve realised the library’s shortcomings, I’m moving towards correcting them. That means removing the psuedo-scheduler from the library, as well as making generic class drivers for both host and device mode inside the library that can be referenced by the user code if desired. Of course, this will not change the underlying API, and those wanting to implement their own classes (like in the case of the DFU bootloader) or use the low level library API will not be affected. However, this will mean that the focus in the demos will be on the demo’s functionality, and not all the USB code around it to get it to work.
Contrast the existing Keyboard device demo from LUFA 090510, with the current in-work version using the HID class driver I’m developing:

#include "Keyboard.h"

USB_ClassInfo_HID_t Keyboard_HID_Interface =
    {
        .InterfaceNumber         = 0,

        .ReportINEndpointNumber  = KEYBOARD_EPNUM,
        .ReportINEndpointSize    = KEYBOARD_EPSIZE,

        .ReportOUTEndpointNumber = KEYBOARD_LEDS_EPNUM,
        .ReportOUTEndpointSize   = KEYBOARD_EPSIZE,

        .IdleCount               = 500,
    };

int main(void)
{
    SetupHardware();

    LEDs_SetAllLEDs(LEDMASK_USB_NOTREADY);

    for (;;)
    {
        USB_HID_USBTask(&Keyboard_HID_Interface);
        USB_USBTask();
    }
}

void SetupHardware()
{
    /* Disable watchdog if enabled by bootloader/fuses */
    MCUSR &= ~(1 << WDRF);
    wdt_disable();

    /* Disable clock division */
    clock_prescale_set(clock_div_1);

    /* Hardware Initialization */
    Joystick_Init();
    LEDs_Init();
    Buttons_Init();
    USB_Init();
}

void EVENT_USB_Connect(void)
{
    LEDs_SetAllLEDs(LEDMASK_USB_ENUMERATING);
}

void EVENT_USB_Disconnect(void)
{
    LEDs_SetAllLEDs(LEDMASK_USB_NOTREADY);
}

void EVENT_USB_ConfigurationChanged(void)
{
    LEDs_SetAllLEDs(LEDMASK_USB_READY);

    if (!(USB_HID_ConfigureEndpoints(&Keyboard_HID_Interface)))
      LEDs_SetAllLEDs(LEDMASK_USB_ERROR);
}

void EVENT_USB_UnhandledControlPacket(void)
{
    USB_HID_ProcessControlPacket(&Keyboard_HID_Interface);
}

void EVENT_USB_StartOfFrame(void)
{
    USB_HID_RegisterStartOfFrame(&Keyboard_HID_Interface);
}

uint16_t CALLBACK_USB_HID_CreateNextHIDReport(USB_ClassInfo_HID_t* HIDInterfaceInfo, void* ReportData)
{
    USB_KeyboardReport_Data_t* KeyboardReport = (USB_KeyboardReport_Data_t*)ReportData;

    uint8_t JoyStatus_LCL    = Joystick_GetStatus();
    uint8_t ButtonStatus_LCL = Buttons_GetStatus();

    if (JoyStatus_LCL & JOY_UP)
      KeyboardReport->KeyCode[0] = 0x04; // A
    else if (JoyStatus_LCL & JOY_DOWN)
      KeyboardReport->KeyCode[0] = 0x05; // B

    if (JoyStatus_LCL & JOY_LEFT)
      KeyboardReport->KeyCode[0] = 0x06; // C
    else if (JoyStatus_LCL & JOY_RIGHT)
      KeyboardReport->KeyCode[0] = 0x07; // D

    if (JoyStatus_LCL & JOY_PRESS)
      KeyboardReport->KeyCode[0] = 0x08; // E

    if (ButtonStatus_LCL & BUTTONS_BUTTON1)
      KeyboardReport->KeyCode[0] = 0x09; // F

    return sizeof(USB_KeyboardReport_Data_t);
}

void CALLBACK_USB_HID_ProcessReceivedHIDReport(USB_ClassInfo_HID_t* HIDInterfaceInfo, void* ReportData, uint16_t ReportSize)
{
    uint8_t  LEDMask   = LEDS_NO_LEDS;
    uint8_t* LEDReport = (uint8_t*)ReportData;

    if (*LEDReport & 0x01) // NUM Lock
      LEDMask |= LEDS_LED1;

    if (*LEDReport & 0x02) // CAPS Lock
      LEDMask |= LEDS_LED3;

    if (*LEDReport & 0x04) // SCROLL Lock
      LEDMask |= LEDS_LED4;

    LEDs_SetAllLEDs(LEDMask);
}

Yep, that’s the whole code, other than the supporting header files and descriptors. As you can see, it all revolves around the new HID class driver, which implements all the backend needed (between the user application and the LUFA library) for the HID USB class. You create a new global of each class driver’s type (for multiple instances of the same class, you make multiple instances of the class driver’s state structure), fill in the required values to match your device’s descriptors, and then link the LUFA events to the class driver functions. Each class driver can also define its own custom events and callbacks for the user application, which is the case here; the HID driver calls the CreateNextHIDReport() and ProcessReceivedHIDReport() functions when reports are sent and received.

This new system means more robust applications, as all the work is in the library itself; as the library is updated, so are all the class drivers. The best part is that it makes it trivial to create compound devices of several interfaces, both of the same class and of multiple different classes.

These changes will be visible in the next release of LUFA, scheduled in the next week or two.

Having a permenent Linux system around (Ubuntu 9.04, on my netbook for those who haven’t read previous posts) has proved to be quite useful for testing LUFA demos simultaneously against Linux and Windows — and all from the one room. A few days ago – just after the recent 090510 release in fact – I started to retest some of the LUFA demos I’d missed. And as luck would have it, I discovered some bugs in the Dual CDC and RNDIS demos. Damn you Murphy!

The issues were identical; I’d missed changing the demos’ control packet handling events over to the new global USB_ControlRequest structure instead of trying to read out the remainder of the control request header at the start of the packet, as was the norm in previous releases. The new global structure is populated by the library internally when the control request is first handled by the library (if possible), and serves to give a unified control header access method between both device and host modes. Fixing those faults were easy, but it was too late for the already released 090510 version. If you’re planning on using either of those demos, grab the latest version from the SVN rather than use the release code.
However, the RNDIS demo remained unhappy on Linux. I’ve tested the (working) RNDIS demo before on Ubuntu 7.10 and 8.04 to little success, but that was perfectly understandable given that the RNDIS class is a Microsoft proprietary standard – and a poor one at that – and the RNDIS handling code wasn’t yet mature. When I first started I had to choose between several evils; either go with the Microsoft RNDIS demo and shut out at the time other OSes, or use the standard CDC Ethernet class which Windows didn’t support. Going by the “support in numbers” philosophy I ultimately ended up implementing RNDIS hoping correctly that it was inevitable that it be incorporated into Linux at some stage.

However, with the new 9.04 release of Ubuntu, containing more mature RNDIS code, I decided it was time to get the demo working under it. Plugging it in got me a fat lot of nothing, although a dmesg showed a few RNDIS related enumeration errors. The error was in the RNDIS request for the “0×00010202″ property, and a quick Google search showed this was a common error.

The Microsoft documentation stated that the 0×00010202 property mapped to OID_GEN_PHYSICAL_MEDIUM, which lets the device indicate to the host what sort of physical medium (Ethernet, WIFI, etc.) the adapter incorporates. Linux wasn’t/isn’t happy about this property not being implemented, even though it is clearly identified as being an optional property.
Not to worry; I simply implemented the property, returning a value of 0 (meaning media unspecified) to the host, and suddenly Linux was happy and everything worked well. I thought it was worth trying to dig up the relevant code and design a patch around it, but it seems I’ve been beated to the punch. The only oddity there is that the patch was supposedly put into the 2.6.26 mainline code, yet on both the Ubuntu 9.04 kernel (2.6.28) and a generic 2.6.30 kernel I tried the same issues cropped up. Either the patch hasn’t be applied upstream, or the patch writer has made an error. I’d love for someone to be able to comment on this further.

On a related note, I tried to make my main laptop dual boot Vista and Ubuntu yesterday, with no success. I’ve tried Ubuntu previously on my laptop and that was disaterous, with it going into thermal shutdown from the Kernel being unable to control the ACPI fan on the motherboard. Trying the latest Ubuntu release gives an instant kernel panic about the wonky BIOS’ ACPI before the bootsplash has even shown. It seems Toshiba really can’t write a decent BIOS…