UBISS2024

This course is designed as a week-long tutorial to engage with ubiquitous devices in the domain of smart environments and how to use machine learning to build smart devices. Here, we use an Arduino Nano RP2040 Connect (https://store.arduino.cc/products/arduino-nano-rp2040-connect).

Link Page

https://www.sketching-with-hardware.org/wiki/UBISS2024-Links

Tasks

Task 0: connect an Arduino Nano RP2040 Connect board

• Install the basic software https://labs.arduino.cc/en/labs/micropython
• connect the board via USB
• Make the orange LED (pin 6) blink using micro python https://docs.arduino.cc/micropython/basics/digital-analog-pins/
• Connect an external RGB LED (pin 2, 3, 4), https://www.sketching-with-hardware.org/wiki/RGB_LED
• Control the external RGB LED (on, off, mix color, brightness)

Solution Task 0.1: LED Blinking

 1 # Blinky example
 2 
 3 import time
 4 from machine import Pin
 5 
 6 # This is the only LED pin available on the Nano RP2040,
 7 # other than the RGB LED connected to Nano WiFi module.
 8 led = Pin(6, Pin.OUT)
 9 
10 while (True):
11    led.on()
12    time.sleep_ms(250)
13    led.off()
14    time.sleep_ms(200)

Solution Task 0.2 Control external RGB

 1 # RGB example
 2 
 3 import time
 4 from machine import Pin
 5 
 6 # RGB LED connected to Nano WiFi module.
 7 ledG = Pin(2, Pin.OUT)
 8 ledR = Pin(3, Pin.OUT)
 9 ledB = Pin(4, Pin.OUT)
10 print("start")
11 
12 while (True):
13     print("*")
14     ledG.on()
15     ledR.off()
16     ledB.off()
17     time.sleep_ms(250)
18     ledG.off()
19     ledR.on()
20     ledB.off()
21     time.sleep_ms(250)
22     ledG.off()
23     ledR.off()
24     ledB.on()
25     time.sleep_ms(250)

Task 1: read Acceleration from Arduino Nano RP2040 Connect board

• read data from the accelerometer and the gyro and print them (Arduino IDE) https://docs.arduino.cc/micropython/basics/board-examples/
• extend you program to write the data from the accelerometers to a file, https://www.sketching-with-hardware.org/wiki/FileIO
• transfer the file to your computer
• optional: add the photo resistors to your board, read their values, and write them to the file, too, https://www.sketching-with-hardware.org/wiki/LDR

Solution Task 1.1: Read Accelerometer and Gyro

 1 import time
 2 from lsm6dsox import LSM6DSOX
 3 
 4 from machine import Pin, I2C
 5 lsm = LSM6DSOX(I2C(0, scl=Pin(13), sda=Pin(12)))
 6 
 7 while (True):
 8     accel_data = lsm.accel()
 9     print('Accelerometer: x:{:>8.3f} y:{:>8.3f} z:{:>8.3f}'.format(*accel_data))
10     gyro_data = lsm.gyro()
11     print('Gyroscope:     x:{:>8.3f} y:{:>8.3f} z:{:>8.3f}'.format(*gyro_data))
12     print("")
13     time.sleep_ms(100)

Solution Task 1.2: Read analog values - Code Example Arduino Nano Connect RP2040

A0 is the analog input with 16 bit resolution. It reads the analog value every second and print it to the console-

 1 #Example usage for Arduino Nano
 2 from machine import Pin, ADC
 3 from time import sleep
 4 
 5 analogPin = ADC(Pin(26))
 6 
 7 while True:
 8   analogVal16 = analogPin.read_u16()
 9   print(analogVal16)
10   sleep(1)

Task 2: Jupyter Notebook

• connect the board
• install the Juypter Notebook, https://www.sketching-with-hardware.org/wiki/Jupyter
• read the accelerometer and the gyro and show it in the notebook

Task 2.1: is it moved?

• read acceleration and gyro
• calculate the differences between values
• show an ouput when it is move
• create a file on the device that logs, when it is moved

Task 2.2: it was turned upside down?

• read acceleration and gyro
• make a rule based "AI" that records
  • it was put upside down
  • it was turned 360
  • it was moved "quickly"

Task 3: ML on Arduino Nano Connect RP2040

We will use https://github.com/eloquentarduino/everywhereml to detect the same gestures as in Task 2.2. For this, install everywhereml:

pip3 install -U everywhere

Using everywhereml we can train a model on a more powerful machine for deployment on a microcontroller. See https://eloquentarduino.com/posts/micropython-machine-learning for example for such a training process. Assuming that our ML model is trained and stored in variable clf then we can save the model to a file using

clf.to_micropython_file("MyModel.py")

The MyModel.py file can then be saved and called directly on the microcontroller. To run the model on the microcontroller, assume your data is stored in x and you trained a RandomForestClassifier. Then you can predict via the following code snippet

import MyModel
clf = MyModel.RandomForestClassifier()
clf.predict(x)

Task 4: connect both boards to WIFI

• connect both boards to WIFI using Tutorial_Network
• use the Arduino Nano RP2040 Connect as output (showing a color)
• use the Arduino Nano RP2040 Connect as input (recognize with rules 3 gestures)

Links

See the full list of links: UBISS2024-Links

Local Links

https://ubicomp.net/sw/db1/var2db.php? http://localhost:8888/notebooks/ArduinoNanoRP2040_v01.ipynb http://localhost:8888/doc/tree/create-ML-model01.ipynb

Reading

Required Reading before the course

• Albrecht Schmidt (2020) Interactive Human Centered Artificial Intelligence: A Definition and Research Challenges. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI '20). Association for Computing Machinery, New York, NY, USA, Article 3, 1–4. https://doi.org/10.1145/3399715.3400873 https://uni.ubicomp.net/as/iHCAI2020.pdf (4p)
• Albrecht Schmidt and Kristof van Laerhoven (2021) How to build smart appliances? In IEEE Personal Communications, vol. 8, no. 4, pp. 66-71, Aug. 2001, https://doi.org/10.1109/98.944006 https://www.eti.uni-siegen.de/ubicomp/papers/sl_ieeepc2001.pdf (6p)
• Albrecht Schmidt (2017) Understanding and researching through making: a plea for functional prototypes. interactions 24.3, 78-81. https://doi.org/10.1145/3058498 https://www.sketching-with-hardware.org/files/functional3058498.pdf (4p)
• Huy Viet Le, Sven Mayer, and Niels Henze (2020) Deep learning for human-computer interaction. interactions 28, 1 (January - February 2021), 78–82. https://doi.org/10.1145/3436958 https://sven-mayer.com/wp-content/uploads/2021/01/huy2021deep.pdf (5p)
• Huy Viet Le, Sven Mayer, Max Weiß, Jonas Vogelsang, Henrike Weingärtner, and Niels Henze (2020) Shortcut Gestures for Mobile Text Editing on Fully Touch Sensitive Smartphones. In: ACM Trans. Comput.-Hum. Interact., vol. 27, no. 5, pp. 38. https://sven-mayer.com/wp-content/uploads/2020/09/le2020shortcuts.pdf (38p)
• Judith Hurwitz, and Daniel Kirsch (2018) Machine learning for dummies. IBM Limited Edition 75, 9780429196645-6. https://www.ibm.com/downloads/cas/GB8ZMQZ3 (Pages 3-18 and 29-47, this is Chapters 1 and 3) (35p)
• Chris Garrett. MicroPython: An Intro to Programming Hardware in Python https://realpython.com/micropython/ (14 pages)
• MicroPython Basics https://docs.arduino.cc/micropython/basics/micropython-basics/ (5 pages)

Recommended Reading before the course

• John D. Kelleher (2019) Deep Learning. https://mitpress.mit.edu/9780262537551/deep-learning/
• Yuli Vasiliev, Python for Data Science: A Hands-On Introduction, https://nostarch.com/python-data-science
• Tutorial on Jupyter Notebooks: https://www.datacamp.com/tutorial/tutorial-jupyter-notebook
• Alex Smola, and S. V. N. Vishwanathan (2008) Introduction to machine learning. Cambridge University, UK 32.34. https://alex.smola.org/drafts/thebook.pdf

Random Commands

pip install micropython-lsm6dsox

picotool.exe load -x C:\Users\ru42qak\AppData\Roaming\OpenMV\openmvide\firmware\ARDUINO_NANO_RP2040_CONNECT\firmware.bin

pip install jupyterlab

pip install everywhereml

python -m pip install jupyter

git clone https://github.com/goatchurchprime/jupyter_micropython_kernel.git

pip install -e jupyter_micropython_kernel

python -m notebook

python -m jupyter kernelspec list

C:\Users\ru42qak\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\jupyterlab>pip install -e jupyter_micropython_kernel

C:\Users\ru42qak\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\jupyterlab>python -m notebook