Building UI for Autonomous Vehicles

Logo for Merlin LabsMerlin Labs


Testing and validating autonomous vehicle behavior is challenging. Doing so while the vehicle is 10,000 feet in the air is even more challenging.

Research

At Merlin Labs, I have been leading design and product development to achieve safe and efficient flight testing. The primary goal was to develop a simple and powerful interface that pilots can control from a tablet. Leveraging web technologies allowed for more rapid feature-development, reliabilty from a stable and known ecosystem, and having a flexible runtime environment (BYOB - Bring Your Own Browser).

I conducted usability interviews with our test pilots which helped greatly understand their needs and wants. Aside from monitoring the state of the aircraft (data such as altitude, airspeed, etc. that a Primary Flight Display would give you), pilots had to perform a fixed set of actions:

  • Change the flight mode (for different levels of autonomy)
  • Perform a set, pre-defined maneuver
  • Tune parameters
  • Check diagnostics/health

Given that these actions were mostly distinct from each other, I designed the UI to maximize each interaction area. This allowed pilots to access parts of the application more reliably, since wind/turbulence was unlikely to cause them to press a wrong button.

I designed a design system in Figma which helped the component library design. Before starting implementation, I designed an interactive wireframe and presented it to pilots to gather feedback. One interesting insight was the importance that color had - especially in cockpit instrumentation. Colors such as yellow/orange/red had to be used with clear intent, since the distinction between each color could communicate very different messages - from a slight performance degradation (that may be ignored) to an impeding danger.

Implementation

I built this tool with the following tech stack, which was quite enjoyable to use: Next.js, TypeScript, and Chakra UI. I built the component library and documented it using StoryBook, a visual documentation tool which makes it easier to preview a components utility and scope. I configured a testing suite using Jest for unit-tests and Cypress for end-to-ends testing.

Constant communication (especially listening!) with the pilots was crucial in ensuring the product fit all and every need. To share the ins and outs of the product, I generated and shared a user manual showing all of the product's functionality.

In Action

The flight test product finally took flight, after 7 months since beginning to work on the project. The first flight was uneventful - and that was the best news to hear.

One highlight was being able to operate the product from the backseat of a tandem-seat canard aircraft. It's not every day you get to battle-test your product while flying!

I hear the product is still going strong after 150+ flight hours...✈️