Home Swear Jar

Having a go at full stack IoT development

Measured by 2 Arduino Nano 33 BLE devices, one in the kitchen, one on the coffee table

About

After completing the temperature sensor project The Temperature At My House, I wanted to venture into the realm of AI and microcontrollers aka tinyML. A home swear jar seemed like a fun use case. The solution design is straightforward. There is an Arduino Nano 33 BLE Sense in the kitchen and one in the lounge room. They are 'listening' all the time.

Nano and 9V Battery

The Nanos are not recording and storing audio. It is processing all sound as a continuous stream, then subjecting it to a predictive model. If the probability of being a swear word is greater than 0.95 (out of 1 so 'highly probable') then the label of the sound and a timestamp is being sent via Bluetooth to a Raspberry PI computer. The PI then uploads the data to a Firebase server. With Bluetooth, the Nano needs to be within 10 metres of the Raspberry. I originally planned to power the units via a 9V battery but the battery did not last a day probably because it is always listening and processing sound. Instead, they are plugged into the wall.

The Architecture

The development process was the usual two steps forward and then getting stuck for hours when the recipe hit some technical roadblock. I re-used my existing Firebase and d3 knowledge and relied heavily on the book TinyML as a starting point. The book is quite good albeit in the four years since being published, the default reference project will not work straight off the page. I also had to learn a bit about Bluetooth communication.

Task 1 was to deploy a basic "wake word" solution trained in Google Collab (an online analytics tool) with sample audio provided by Google. The model can then be transferred to the Arduino device using a special tinyML code library. Turns out that the library is no longer fully compatible with Tenserflow 2.0. The firmware would simply not upload. The fix was simple but difficult to solve since the error message relates to memory but the actual solution is something else¹. So far so good.

Task 2 was to record my own audio and train a custom model in Google Collab. I recorded the audio on my phone and followed the cleaning steps described in the tinyML book. The model training itself successfully completed. I could get an output in Collab and transfer it into the Arduino code editor. However, the firmware failed again to upload to the device. Quite strange since any other model trained on the preloaded audio worked fine. After hours and hours, I was still not able to debug so instead tried the Edge Impulse service for training a custom model and deployment. This worked really well and I can highly recommend it.

I had 20 examples of each person swearing and re-used some other words and background noise from the Google library. Even with such a small volume of examples, the model is suprisingly OK, but could be better. Here is the confusion matrix on the test set (examples os audio the model had not yet seen):

Test Set Confusion Matrix

A confusion matrix shows the percentage of cases classified correctly (Edge Impulse chooses whichever class has the highest probability), as well as the the degree of misclassification. If you read the table row-wise, the model did well with my F-bombs (M-F) but not so well with Mum's F-bombs (N-F) and my S#!t (M-S).

The Edge Impulse UI also has some good tools to inspect examples that are misclassified (see below). We can see if the model itself is just poor quality or that the example is a bad audio recording and thus difficult to classify.

Inspecting Examples

If I had more time..... the main area for improvement is the speech detection model itself. And there are three major ways to improve:

Model training design
More data
Tuning hyperparameters/model architecture

Model Training Design
You might not have picked this up but there is a design flaw in how I've set up the model training. I want the AI to be able to distinguish my swearing from mum's swearing but I have not added other people's cussing.The "other words" mentioned above are just other people saying common English words. If the sensor is next to the TV and it's playing The Wire then there's a good chance of falsely identifying our swearing. Ideally, I'd need to collect a dataset of other people swearing too.

More Data
My model is only based on 20 examples of each swear word for each speaker plus some other examples for other words and noise. But even with this small amount, accuracy was OK. Perhaps even 10 more examples per speaker/word would likely make a difference, perhaps using the Arduino device itself to capture the audio.

Tuning Hyperparameters
Even with a small number of examples, Google Collab took 90 minutes to train a model with a GPU resource using the default hyperparameters/model architecture. Thankfully it was much faster with Edge Impulse, around 10 minutes. With a trial and error approach to testing different model architectures, there might be some gains in accuracy.

I'd also like to test the solution with an analog microphone rather than use the built-in digital microphone on the Arduino, to see if I can get better detection. A classification solution is only as good as the quality as the data captured. But the Edge Impulse firmware is written for a digital audio signal. I would have to get deep into the weeds of signal processing to modify it and create a version that works with analog audio.

One more tip. The tinyML book has test projects for different devices. I had to wait several months for the Arduino BLE 33 Sense to come back in stock at my local retailer. In the interim, I tried to make do with a Sparkfun Edge. Unfortunately, the device simply failed to work. It would not communicate with my Macbook, iMac or Raspberry PI. I could not upload any firmware to the device. Forget it. Huge waste of time.

¹ The solution for getting Tensorflow 2.0 models to work with the tinyML Arduino library: in the main Arduino .ino file, replace

if (micro_op_resolver.AddDepthwiseConv2D() != kTfLiteOk)

with

if (micro_op_resolver.AddConv2D() != kTfLiteOk)


      

    TLDR: Having a go at full stack IoT development. 
     michael@thisdatatalks.com