In a previous update, we introduced our work on Packetry, our new front-end software for using Cynthion to capture and analyze USB traffic on the wire. In this update, we’re going to talk a bit more about the design of that software and explain some of the work we’re doing to make it as fast and easy to use as possible.

The Need for Speed

One of the most exciting features of Cynthion is its ability to serve as a protocol analyzer, letting you intercept and capture USB traffic on the wire, in the field, from real devices.

Fully benefiting from this feature requires the help of some very efficient software on the capture host. A high-speed USB capture, including metadata, can generate over half a gigabit per second of raw data. To see what’s happening in real time as the capture progresses, all that data needs to be processed at least as fast as it’s captured, which is close to 50 megabytes per second.

In fact, real-time performance isn’t really enough in practice. When opening a saved capture from a file, the software needs to process packets many times faster than real time. At real-time decoding speed, a busy minute-long capture would also take a whole minute to load. So we need a solution that can process USB packet data many times faster than real-time speeds, which means a throughput of hundreds of megabytes per second at least.

Because of these requirements, we’ve designed Packetry from the ground up to achieve the fastest possible decoding speeds and to scale seamlessly to captures of unlimited size, bounded only by disk space. We’re also thinking ahead: Packetry has been developed for USB 2.0 analysis with Cynthion, but in the future we may want to use it to analyze higher speed protocols.

All these factors have made performance critical to success: so how did we achieve it?

Laziness as a Virtue

To achieve the speed and scalability we need, we must do the minimum work necessary for each packet at capture time. We don’t need to fully interpret all traffic as it’s captured: we just need to follow each packet’s effect on the protocol and store things in a form we can efficiently access later.

Rather than constructing individual data structures to represent packets, transactions and other protocol elements, we simply write all captured packets out into a flat bytestream in order. In this form, the capture can be very efficiently written to a file as it progresses.

As we do so, we build a set of indexes which describe where to find the start of each packet, the start of each transaction, and so forth. Those indexes are how we describe the protocol structure. They are designed to provide just enough information for the UI to later look up any part of the capture, decode it on demand, and display it.

Each index is a monotonically increasing sequence of integers. Because of that, we can store them very efficiently. We use a simple compression scheme to minimize the storage overhead of the indexes. To ensure scalability, these indexes are also streamed to files themselves. As such, capture size is limited only by available storage, never by available memory. All data structures in memory have a bounded size which is independent of the size of the capture.

In this approach, we gain scalability by giving up a little speed at display time. This is a good tradeoff to make, because, unlike capturing or loading data which may happen at extremely high speeds, displaying data is constrained by the bandwidth of the human user. There can only be so many things on screen at once, and the user can only browse through them at human speeds. We do not have to make rendering the display very fast for it to feel instantaneous.

Traffic Display

USB is a highly structured protocol: packets are grouped into transactions, transactions into transfers, and transfers are attached to specific endpoints on devices. Our UI displays traffic hierarchically according to that structure, making captures easy to understand and explore. A similar design approach was pioneered in ViewSB, but in Packetry we’ve now made it fast and scalable to large high-speed captures.

Our GUI has been built on GTK 4, which has built-in support for displaying large lists and trees by lazily loading only the parts currently visible on screen, recycling UI widgets for efficiency, and preloading ahead of scrolling. When you scroll through the traffic view in Packetry, the packets required are loaded on demand using the capture indexes, decoded on the fly, and used to generate the summaries you see of packets, transactions and transfers. All this happens live, too: if you’re running a capture, you’ll see new traffic appear, and the descriptions of existing items may be updated as further packets come in. When you load a capture from a file, you can start exploring it immediately, even while the packets later in the file are still being indexed.

Threading Model

It’s been a while since individual CPU cores got significantly faster; these days performance gains usually come from parallelization to take advantage of multiple cores. However, some tasks just can’t be parallelized, and the only option is to make them as fast as possible on a single thread.

When analyzing packets captured on the wire, every packet matters to the overall state of the protocol. The need to deal with invalid packets and protocol errors means it’s not possible to make assumptions about structure. Interpreting traffic correctly requires looking at every packet one by one, in order, and updating the protocol state at each step. That means our packet decoder has to run as a single thread and be highly optimized for throughput.

We can, however, move everything else out to separate threads so that the core decoder can run as fast as possible. Packetry runs as three threads, each feeding data to the next:

1. The capture thread deals with streaming captured packets from a Cynthion device over USB.

2. The decoder thread processes captured packets, stores them and builds the capture indexes.

3. The UI thread runs the user interface, reading from the indexes and stored packets to display the captured traffic in a human-readable view.

A key feature of the design is that all the interactions between these three threads are lock-free: they cannot block each other. The capture thread supplies packets to the decoder thread through a FIFO queue. The decoder and UI threads use our own lock-free stream implementation, which allows a single writer and any number of reader threads to efficiently share a growing data stream whilst it is being buffered and written out to storage, then mapped back into memory as needed.

Keeping these threads decoupled from each other helps us ensure that capture will always continue to run at a consistent throughput, no matter how complex the the traffic is or what analysis is being done in the UI.

Head to Head

So how fast is it? To give a quick illustration, here’s Packetry and Wireshark loading the same file, side by side. The file is a 300MB capture in pcap format of a HackRF in use.

Packetry finishes loading the file 10x faster — but you don’t even need to wait for that to happen to start exploring the capture. The view is ready to interact with as soon as the first packets are decoded. That happens almost instantly, and you’re up and running immediately with a fully hierarchical, human-readable view of the traffic.

Trying it out

Packetry is still in active development and we don’t have releases or binary downloads yet. If you’re really keen to play with Packetry you can build it from source and try it out with the example capture files that are included in the repository for testing. Build instructions are in the repository.

In December of 2022, we published a post about the HackRF One shortage and the hardware revision our engineering team completed so that we could continue manufacturing HackRF One. This hardware revision was necessary because we had difficulty sourcing critical components during the global chip shortage, mainly MAX2837- the RF transceiver IC used in every revision of HackRF One before r9. At the time of that post, we had a significant backlog of orders, and we were uncertain about how long production would take with COVID-19 slowing down operations at the factory in China. Today, we have good news: production of r9 went very smoothly, and the finished HackRF Ones started shipping to our warehouses in late February. As of now, all of the backorders for HackRF One have shipped to our resellers.

With over half of this last production sold and shipped due to the backlog, we are already preparing for a second 2023 production run in quick succession. Some lessons we learned from this HackRF One shortage are to invest in components early for products we know we want to keep producing and that components on the shelf are preferable to components on order if we can find them in stock at a reasonable cost. Fortunately, we found more MAX2839s (the substitute component for MAX2837 in r9) with good date codes and purchased them for a second production of HackRF One r9. We have also already purchased production quantities of several other HackRF One components that have very long lead times, like the clock generator chip, the CPLD chip, and the microcontroller. These components are on the shelf and ready for us to use in the upcoming production, so we don’t have any immediate concerns about unreliable chip distributor lead times impacting our production schedule. Most likely, it won’t be as easy to find more MAX2939s for future productions because that part is obsolete. That means there will only be one more production of r9, and subsequent hardware revisions will use MAX2837.

We are already ordering components with long lead times for HackRF One productions as far out as 2024. Before the chip shortage, it was entirely realistic to acquire all the components needed for HackRF One and complete a production run within six months. That is no longer practical because even though purchasing conditions are improving for some components, chip destributors are still quoting lead times of up to a year for several key HackRF One components. MAX2837 availability in particular remains scarce; the average lead time has tripled and the manufacturer price has nearly doubled. The MAX2837s we ordered in 2021 for Fall 2022 production have yet to arrive, although we do expect them to be delivered finally this summer. We plan to use those in a third 2023 production near the end of this year. We don’t know when (or if) things will return to normal with the chip market. So to prevent future production bottlenecks caused by one or two missing essential components, we will continue to plan manufacturing schedules and component orders further ahead of time.

We appreciate our resellers for their cooperation while we navigated the challenges presented by the global chip shortage, and their customers who waited patiently (some for months) for their HackRF One orders to be delivered. Thank you for being so supportive! We can’t understate this: you are the reason we are still here!

The February recipient for the Great Scott Gadgets Free Stuff Program is Mihajlo. Mihajlo is a student in Serbia who will be studying Electrical Engineering in college. He plans to use the HackRF One we are sending him to teach people in his community about SDR, RF, LRPT images from NOAA, and other fun signals. Mihajlo is also an open source contributer to projects focused on providing simple scripts and instructions to set up low-cost base stations. We are proud to support Mihajlo in his projects as he gives back to his community in so many ways!

The January recipient for the Great Scott Gadgets Free Stuff Program is Guillermo. Guillermo is from Spain. He had a career in computer security about ten years ago before switching to another career. He’s back now and excited to use the HackRF One we are sending him to explore interesting topics like RF, Lora, Zigbee, and BLE. We are happy to support Guillermo on his journey back into computer security.

Note: This is a crosspost of a Cynthion update on Crowd Supply: https://www.crowdsupply.com/great-scott-gadgets/luna/updates/renaming-luna-hardware-to-cynthion.

Until now, the name LUNA has referred both to our USB multitool hardware platform and to the USB gateware framework that supports it. From now on, we want these projects to have separate names. We’ve decided to name the hardware platform Cynthion. The gateware framework will continue to be called LUNA.

Why are we making this change?

This renaming is happening because we have often had conversations, both internally and with external folks, in which we’ve had to say, “Wait. Are you referring to LUNA hardware or LUNA gateware?” Additionally, we’ve repeatedly fielded questions from folks who have been confused about different features supported by the gateware framework vs. the hardware platform. One common point of confusion in conversation is that LUNA gateware supports SuperSpeed USB while the Cynthion hardware does not. Lastly, projects like ORBTrace and Amalthea use the LUNA gateware framework to drive their projects, but they do not benefit from the association with Cynthion. It would be nice for these projects if they could indicate that they are supported by LUNA gateware without making them seem like add-on boards for Cynthion. Meanwhile, Cynthion is no longer just a development platform for the LUNA gateware, it is its own product and we need a name to refer to just the hardware while in conversation.

What happens next?

In the next few weeks, you can expect to see the hardware design files move from the LUNA repository into a new Cynthion repository we will create in the Great Scott Gadgets organization on GitHub. Following that, you will see the hardware documentation migrate into a folder in the Cynthion repository. Before you receive your hardware, Cynthion, its enclosure, packaging, and marketing materials will be updated to reflect the new name.

All LUNA gateware framework materials will remain as they are in the LUNA repository. There should be no need for existing projects to change how they are interfacing with LUNA.

Why the name Cynthion?

When choosing a name for this hardware platform, we wanted to pick something moon-related like LUNA. As we scrolled lists of moon-related words, “apocynthion” and “pericynthion” stood out to us. We played with subwords like “pericynth” but as a team, we eventually settled on Cynthion. The word Cynthion means “related to the moon” and is derived from Cynthia, an alternative name for Selene, the Greek personification of the moon.

The December recipient for the Great Scott Gadgets Free Stuff Program is Shiva. Shiva is new to hardware and IoT. We have sent Shiva a GreatFET One so they can do some research on devices they have in their own home. We are looking forward to hearing about their results!

Note: This is a crosspost of a Cynthion update on Crowd Supply: https://www.crowdsupply.com/great-scott-gadgets/luna/updates/luna-revision-0-5

We are happy to say that hardware revision 0.5 for LUNA has been completed and that revision 0.6 is well under way! Some highlights from revision 0.5 are an upgrade to using KiCAD 6, adding some of our planned part substitutions, and improving labeling. Full details of hardware revision 0.5 can be viewed in the GitHub pull request for that task https://github.com/greatscottgadgets/luna/pull/190. As for hardware revision 0.6, our current set of tasks can be seen in this GitHub issue: https://github.com/greatscottgadgets/luna/issues/185. Progress has been made on almost every item in the r0.6 checklist and we will check the items off as they have been finalized and put through our review process. We look forward to giving you more updates soon!

Once in a while, we get emails from people in the community who want to know what they can do to increase their chances of getting hired at Great Scott Gadgets (GSG) when we do happen to open up a position. We really appreciate this enthusiasm! Below we outline some of the skills and topics we assess when hiring someone to join our small, completely-remote team.

1. Know our products and projects. Our products are listed on our website. The Software and hardware details for our products and projects are in the repositories on our GitHub. When we interview we ask which of our products you are familiar with, if there are any you are excited to work on, and if you have experience with any of them. While it is not required for an applicant to have used our products, it really helps when applicants have an idea of what our company does and what they would like to do when they join us. Yes, we often hire with a specific project in mind, but we do want to make sure the position fits the person and can often make adjustments.

2. Have skills in the tools and languages we use. Take a look at our GitHub repositories and see what technologies and programming languages we use. We are a company dedicated to open source and most of our development happens in the open by team members forking a repository, making changes in a branch, and then submitting a pull request just as anyone in the community is able to do. We do like to see GitHub contributions from new team members within the first few weeks of working at GSG so it is important to know at least one of the languages we work with well enough that you can jump right in.

3. Contribute to open source or build a project to talk about in interviews. Our interviews do not have any whiteboarding, homework, or any other excessively long skills testing process. We currently assess applicants on two phone screens and on code, projects, documentation, and writeups you have posted publicly. We do understand that some of the people eager to join GSG are coming from a career filled with NDAs or restrictions on personal projects and contributing to open source. If you don’t have public materials for us to review, please let us know and we’ll work with you to make accommodations.

If you have more questions about working at Great Scott Gadgets or our hiring process, please reach out to careers@greatscottgadgets.com.

After 12 years and 17 production runs, Great Scott Gadgets is retiring our first product, Ubertooth One, from our hardware catalog.

GSG’s founder Michael Ossmann designed Ubertooth One because he wanted a device that could detect and monitor Bluetooth. At the time, such instruments existed but cost at least five figures—prohibitively expensive for most security researchers. His goal was to design an open-source, affordable-to-make tool that anyone in the security community with basic soldering skills could assemble. At the project’s inception, his intent was not to sell hardware but to provide a solution to a problem that no one else had solved. However, demand from the community prompted him to start GSG and launch a Kickstarter campaign that funded the first production.

Ubertooth One enabled more than starting a company; it became an essential part of the wireless security professional’s toolkit and aided research that improved Bluetooth security and function. One notable example is Mike Ryan’s Bluetooth Low Energy (BLE) security research. Through this work, Mike contributed BLE capabilities to Ubertooth and became a core developer of the project. More recently, Ubertooth One was instrumental in research into Apple’s Continuity protocol presented by Sam Teplov at ShmooCon in January 2020. Over the years, Ubertooth has equipped researchers to improve the Bluetooth protocol’s function and reverse engineer countless Bluetooth devices and even non-Bluetooth 2.4 GHz wireless systems such as electric skateboards. Talking to Michael this week about his journey with Ubertooth, I learned of an encounter at a conference in Asia where a stranger approached him and said “Thank you for Ubertooth. I couldn’t have done my Master’s thesis without it.”

At the time Ubertooth One was designed, BLE didn’t yet exist. The protocol now known as Bluetooth Classic was the only Bluetooth protocol. It was common for Bluetooth devices to operate in non-discoverable mode, making them invisible to all but the most expensive monitoring tools. Ubertooth One made it possible to detect and identify non-discoverable Bluetooth devices, an essential function for wireless security practitioners and researchers. Today, most Bluetooth devices use BLE rather than Bluetooth Classic, and several low-cost options are available for monitoring BLE. For more esoteric capabilities, including the detection of non-discoverable Bluetooth Classic devices, researchers can use Software Defined Radio platforms such as HackRF One to implement the same functions as Ubertooth. Even though Ubertooth is still a valuable and widely adopted tool, it is no longer the only option.

When the global chip shortage struck, our small team faced difficult choices about which products to redesign for available components. After considering changes in the Bluetooth landscape, the amount of redesign effort required, and the work cycles available to our team, we decided it was time to retire Ubertooth One. Consistent with our mission, we will continue to prioritize making and maintaining tools that, like Ubertooth in the early years, allow innovative people to do things they haven’t previously been able to do.

Even though we are now sold out of Ubertooth One, you may still be able to buy a unit made by GSG while reseller stock lasts. The Ubertooth project is open source, so if you can’t purchase an Ubertooth One, you are welcome to use the design files in the project repository to make your own. We will continue to monitor the repository for issues and pull requests, but we have no plans for hardware or software enhancements.

The Ubertooth project has meant a great deal to Great Scott Gadgets, and we’d like to sincerely thank our users, our resellers, and all the people who have supported us and contributed to the project over the years for coming on this journey with us. Special thanks to Dominic Spill, who started gr-bluetooth, which was foundational to Project Ubertooth; Jared Boone, who mentored Michael in the original hardware design; and Mike Ryan, who made significant contributions to the project. If you have any stories you’d like to share about Ubertooth One, please come tell them in the Great Scott Gadgets Discord server or email us at info@greatscottgadgets.com.

The November recipient for the Great Scott Gadgets Free Stuff Program is Alex of the CCCSBG Hackerspace. A group of people at the CCCSBG Hackerspace are making an effort to explore the protocol spoken between ancient IBM3270 terminal equipment and their controller. Alex points out that Andrew Kay has done similar work for text-only traffic [1], but mentions that his hackerspace wants to chip away at the graphics capabilities of the IBM 3270. We have sent Alex and CCCSBG a GreatFET One so they can sniff the traffic that is going through the NS DP8340 and NS DP8341 chips on the devices.

[1] https://ajk.me/building-an-ibm-3270-terminal-controller