Skip navigation
2014

Can anybody tell me how to configure an SD card, wifi-shield, and the serial monitor to all work simultaneously??? I'm trying to capture an image using motion detection and, afterwards, save the array of 100 images to the SD card.  Afterwards, I hope to use the wifi-shield to push a notification to an app from OpenCV code. Is this possible?

Everything that happens on a micro-controller has to be known. This includes dealing with memory. Many people (myself included) have developed a wonderful set of code that can dynamically use and free memory through the standard malloc and free functions. Only later do we discover that our available memory is getting reduced as time goes by -- followed by undetermined and random failures.

 

This is because no microcontroller (the ArduinoArduino included) has the ability to garbage collect.

 

For instance, let's say you malloc A and then malloc B. You then free A. There is now a memory hole (the size of A) that will never be closed again. Essentially a non-existing A is now taking up memory. If you now malloc A again, it will not be stored in that hole -- new memory will be taken up. You can see how this can become a problem.

 

To solve this problem, I developed the library ReMem (Reusable Memory), located in my usertools library, which among other things contains a full featured User Interface.

 

Documentation of ReMem can be found here. Basically ReMem creates an object (with a specified size) that has two functions: rmalloc and free. These work exactly the same way as the conventional malloc and free except when you free data, it will be used again if the same size data is requested. (The disadvantage is that it can be slow, and takes an extra byte of data per malloc).

 

 

Upcomming

The code is currently stable, but I'm going to be adding the ability to reclaim data that has been freed at the end, as well as cannibalize large allocations. Stay tuned.


Update

I have a new, more tested library for this purpose called tinymem. Check it out here: https://github.com/cloudformdesign/tinymem

User interfaces can take up a lot of memory. That is why, while developing the Arduino User Interface I focused on developing a way to store things in Program Memory instead. It turns out that storing -- and then using -- program memory objects can be slightly confusing. So I thought I would write a quick post about it.

 

Most people are familiar with the simple use of program memory for strings. Instead of writing

 

Serial.println("This is taking up a lot of mem");

 

You can write

Serial.println(F("This is taking up no mem"));

The F() stores the string in Flash (or Program Memory) instead of creating a character array in RAM and then passing it to println. This is very good, considering that the Arduino UnoArduino Uno has only 2k of SRAM and more than 32k of Program Memory!

 

Storing in Program Memory

But what if you want to do something more? The threading module requires an array of pointers to functions. At first glance, this also looks fairly easy.

typedef int (* funptr)(int index)  // declare the funptr type

 

int return_0(){

  return 0;

}

 

int return_1(){

  return 1;

}

 

PROGMEM const funptr fptrs[] = {&return_0, &return_1, NULL} // fun_x are declared functions

The problem comes about when you have to pass the pointers variable to other functions -- there is no way to declare that it is coming from program memory!

 

Before I explain, perhaps it is a good idea if we remember what a pointer is: it is a number which represents a certain location in memory. So:

 

int x = 8;          // x is storing the integer 8

int *p_x = &x;  // p_x is storing the location of x

 

So right now our fptrs are just an array of data that is stored in Program Memory, each of them pointing to functions.

 

Lets say we have a function like the one below:

 

int call_func(int index, const TH_funptr *ptrs){

     return ptrs[index](); // THIS DOESN'T WORK

}

 

The reason the above code doesn't work is because fptrs are in program memory! To access it we have to read out of program memory. Fortunately, there is a function to read 2 bytes of data (a word) from program memory: pgm_read_word

 

To solve this problem, I developed the get_pointer routine.

 

void *get_pointer(PGM_P s, uint8_t index, uint8_t size){

  return (void *)pgm_read_word(s + index * size);

}

 

So now our function would look like

 

// not very useful, but at least it works

int call_func(int index, const TH_funptr *ptrs){

     return get_pointer((PGM_P)fptrs, index, 2)(); // This works. Note that you have to cast fptrs as type PGM_P

}

 

 

More Complex Storage in Program Memory

 

The Arduino page has an excellent tutorial on how to store strings in program memory, and that is what is fundamentally used in the usertools library. Basically, you must construct each pointer, and then store an array of pointers (tons of fun).

 

This also applies to more complex data types. I would have THOUGHT that you could store basic types in an array, but it turns out you can't. For instance, variables are composed of a pointer to the variable and an 8 bit unsigned integer representing it's size. To store a variable, you have to store a pointer to each element in program memory, and then store the array of those pointers into program memory as well (using way more program memory than I would like, but no RAM).

 

That is the reason to expose variables you have to do:

UI_V(v1, myvar1);              // Variables have to be declared specially. Declare variable names first with UI_V

UI_V(v2, myvar2);              // Variables have to be declared specially. Declare variable names first with UI_V

expose_variables(UI_VA(v1), UI_VA(v2));     // Then wrap the variable names in UI_VA. Alot of things have to be done to take up zero RAM!

 

Now you know what goes into taking up zero RAM.

One of my first big micro-controller projects was called the Neonatal Closed Loop Oxygen System. When babies are born prematurely, they have to be fed very specific amounts of oxygen. Too little oxygen and they suffocate, too much oxygen and they can die from oxygen toxicity. Current methods are not advanced, essentially involving a nurse adjusting the oxygen levels manually. The goal of the project was to develop an automatic system.

 

We had to prevent this little guy from getting oxygen poisoning

450px-Premature_infant_with_ventilator.jpg

The project had the following design requirements:

  • Check oxygen levels from a UART input.
  • Process oxygen levels and respond appropriately (long calculation time).
  • Adjust valves electronically to correct oxygen levels.

 

The project also had to do the following:

  • implement a simple user interface with a few buttons and an LCD screen
  • Be able to sound an alarm for a set amount of time if there were issues.
  • Export periodic debugging data to improve the product.


All of these things had to happen simultaneously, and it was critical that nothing prevented the microcontroller from adjusting the oxygen. It would be no good if the Alarm sounding caused all other functions to cease!


I was in charge a few aspects of the project, but one of the main ones was developing a user interface and threading structure. Essentially my problem was: how do you do all of these things at once? I'm not going to go over the solution I used (it was nasty and difficult to understand), but about a year later I discovered the protothreading library written by Adam Dunkels. This made threading significantly easier, but it was still a pain -- you had to declare each thread object individually, it seemed almost impossible for functions to use multiple threads (as variables have to be static), and it still felt way too much like micro-controller programming (and not enough like Arduino programming!)

 

Why do you need a user interface?

Maybe you aren't trying to save a baby's life, maybe you are trying to figure out how to control a robot like the line follower below. You have to take user input, output debug data, take line readings, and control each motor separately. That's a lot of things for a little Arduino to do!


Line follower robot using an the ArduinoArduino and the  Motor Control ShieldMotor Control Shield

800px-Line_follower.jpg

 

So, a few months ago I started writing a threading and user interfacing library called usertools. Usertools contains the following files:

  • threading.h -- basic and lightweight threading. threading_led_example2 is only 6.8 KB of program memory and 240bytes of RAM!
  • ui.h -- lightweight and full featured user interface. ui_led_example3 and ui_led_example4 take up only about 11.5KB of program space and 320 bytes of RAM to implement a full featured user interface. This is the focus of this post.
  • errorhandling.h -- An error handling and debugging library. This will be the subject of a later post.

To demonstrate the usefulness of the library, this post is going to focus on ui_led_example4 -- as this example demonstrates most of the benefits of the user interface library. Before I go into that, lets overview what has to happen to create a low-memory threading and user interface

 

  • You have to store everything you can into Program Memory (Flash). This is done through the setup functions. For more information on the internals, see my blog post.
  • You want to be able to dynamically create and destroy data. This is done through the pthread class. Also, dynamically allocating and freeing memory is something you are not supposed to do on microcontrollers, so I had to create a class to do it. See my blog post about it.
  • You have to expose the threads and then "handle" them. In other words you have to have a way to know if they are active, deactivated, or should be killed and do the appropriate thing to them. This is done by expose_threads(...), schedule_thread(thread), and kill_thread(thread). Threads are handled from ui_loop() or thread_loop()
  • You have to monitor the serial data and behave appropriately, allowing the user to schedule threads, kill threads, call functions, and set/get variables. This is all done in ui_loop()

 

Upload the code to your Arduino

 

Get the latest release of usertools library from github. Move the "usertools" folder to your Arduino libraries ([Arduino Folder]/libraries). Start Arduino and go to:

file->Examples->usertools->ui_led_example4


Then upload the code to your Arduino. Open your Serial Terminal (Cntrl + Shift + M) and make sure to set the drop-down boxes in the bottom left to "Both NL & CR" and "57600 Baud".

 

Once you have done that, type the following (each followed by enter):

  • “?” to get info on commands you can type.
  • "v sub_time 500" -- you will see the led's blink much faster (500ms is being taken off both periods)
  • "v sub_time" -- get the value of sub_time in hex
  • "v sub_time 0" -- set it back to 0 (slow blinking)
  • "k led1" -- Kills thread led1. This will make it so that only 1 thread is blinking (blinks at period of 900ms)
  • “k led2” -- everything will be killed (no blinking)
  • “reinit” -- calls the reinit function, everything will be set to default values!

 

Explanation

 

Note: There is full featured documentation for all modules located in the Documentation folder (just open the .pdf files). If anything is confusing you, please reference these documents as they should explain most questions.

 

Declaring our Thread function

 

In this part of the code, we include our usertools library <ui.h> and write our main thread function "blinky_thread". As you can imagine, this thread will cause the LED to blink.

 

At first blinky_thread might look a little odd -- that is an awful lot of code to just blink an LED! The key is in the PT_WAIT_MS call -- this call is what makes the whole thing tick. If you run this program, you will notice that your LED is blinking... weird. Sometimes it's a long period, sometimes it is really short. This is because we have two threads running that are toggling the same LED. Well get to that soon!

 

Keeping track of Threads

 

in this part, we label the index names of our threads, functions and variables (we will give them string names later). This is just so that it is easy to get them and schedule them later in the code.

 

Creating the reinit function (accessible through the terminal)

 

This function will be used in our setup -- but it is also declared such that it can be called from the command line. Take note of the function type: uint8_t thread_function(pthread * pt). All functions or threads must be of this type.

 

Let's go through this function step by step:

  thread *th;

We will use this pointer to access our thread's data later in the function.

 

  TRY(set_thread_innactive(get_thread(LED1)));

  clrerr();

  TRY(set_thread_innactive(get_thread(LED2)));

  clrerr();

Remember that this function will be called by both the setup AND by the user. If it is called by the setup, then we wouldn't need to set these threads innactive. However, if it is called by the user we will need to set them inactive before we can re initialize them.

 

The TRY macro is part of the errorhandling library I wrote to make debugging and error checking easier. For more information, open Documentation/errorhandling.html in your web browser.

 

th = get_thread(LED1);

th->pt.put_input(1000);

th = get_thread(LED2);

th->pt.put_input(900);

These lines retrieve the thread objects and then put a value into their input buffers. This sets one thread to toggle the LED at a 1000ms period, and the other to toggle at a 900 ms period.

 

Note that every protothread implements a dynamically allocated linked list. This means that you can add input and output data to the pthread object and then delete it (with clear_data() or del_input(index)). In addition, when a thread is killed or set innactive, all of it's data is destroyed and reclaimed for future use.

 

  schedule_thread(LED1);

  schedule_thread(LED2);

The threads are finally scheduled, and will be called by the ui_loop at the bottom of the file.

 

Setting everything up

 

During setup is probably one of the most confusing parts of this module. It is very important that you do things in the right order and in the right way. Let's take a look!

 

expose_threads(TH_T(blinky_thread), TH_T(blinky_thread));  // wrap threads in TH_T

This line declares the threads that you are exposing. Once exposed, you can schedule or kill threads through the user interface or through code (as you saw above).

This is the only place that you can expose threads. Every "thread" is really a function of the type uint8_t (*TH_funptr)(pthread *pt) (for the newbee, this is a function pointer object. Your function must take one input pthread* and return uint8_t)

 

Note that all of this is done for you by wrapping each function name with TH_T.

 

expose_threads simultaneously allows you to schedule the thread, as well as kill (or schedule) the thread from the command line. For example, because of the names we are creating below, LED1 can be killed with "k led1" and then rescheduled with "t led1 PERIOD", where period is the blinking period you want (an integer).

 

expose_functions(UI_F(reinit));    // Wrap functions in UI_F

The same concept as expose_threads (except you use UI_F). This allows you to access the function reinit from the command line. Because of our naming scheme below, this function can be called with simply "reinit"

 

Note: Functions that are called from the command line only run once, whereas threads will be called every loop (until they return PT_EXITED or are killed)

 

UI_V(v1, sub_time);              // Variables have to be declared specially. Declare variable names first with UI_V

//UI_V(v2, othervar) -- if you had more variables, continue in this way

expose_variables(UI_VA(v1));     // Then wrap the variable names in UI_VA. Alot of things have to be done to take up

                                 // zero RAM!

 

Exposing variables is a little more involved, and requires you to pre-set a name to each variable, and then expose the names by wrapping them in UI_VA. If you are wondering why this has to be the process, then please see my blog post about it.

 

Because of our naming scheme, we will be able to access this variable from the terminal with the command "v sub" or set it with "v sub VALUE"

 

// The names have to be done similarily to variables

UI_STR(tn1, "led1");

UI_STR(tn2, "led2");

expose_thread_names(tn1, tn2);  // no wrapping required here.

 

UI_STR(fn1, "reinit");

expose_function_names(fn1);

 

UI_STR(vn1, "sub");

expose_variable_names(vn1);

Exposing names is done similarly to variables.

 

Note: if you are not using functions or variables, you have to explicitly declare this with no_functions() and no_variables(). See this in ui_led_example3

 

void setup() {

  pinMode(LEDPIN, OUTPUT); // LED init

  Serial.begin(57600);

 

  Serial.println("\n\n#########################  Start Setup");

  Serial.println(freeMemory());

  // This makes the names actually accessible to the threading module.

  set_thread_names();

  set_function_names();

  set_variable_names();

 

  // this must be called before any threading (except setting names)

  setup_ui(200);  // 200 == the amount of dynamic memory our threads can use.

                  // I recommend at least 100, more if you use alot of strings

 

  // This has two purposes: it initilizes the values during setup, and it can be called

  // by the user to reinitilize the values.

  reinit(NULL); // Note: NULL is only used because the function doesn't actually use it's pthread input.

 

}

 

In our setup function we call the set_NAME_names() to expose the names to the system, and then call setup_ui(200).

 

setup_ui(mem) must be called before anything is done with threading or the user interface (except setting the names). The "200" is the amount of memory we have set asside for dynamic memory. This memory is used for all of our pthread put_input and put_output data. If you are using strings, I recommend using at least 200 bytes. For more information about dynamic reusable memory see my blog post about my ReMem module.

 

 

Call in the loop

Now the only thing left is to call one function during the loop:

void loop() {

  ui_loop();

}

ui_loop takes care of everything for you -- reading the user input, putting into the threads pthread objects so you can get the inputs, and killing threads.

 

Summary

 

As you can imagine, this library took a significant amount of work to complete. It is currently in Beta because it does not have enough testers and is not currently used in any project. I ask the Element14 community to help me with the following:

  • Are you working on an open source project that could use threading and user interface functionality? If you are, please let me know if you would like help implementing this library in your project. I would love to be part of the team!
  • This library needs additional testers, and developers would be much appreciated as well! There are a few features I am planning to implement in the next couple of months, like Signal blocking and a better thread-spawning system. My hope is that this will eventually become a standard Arduino library, and to make that happen I could use other developers!

If you are interested in either of these, please contact me at Garrett@CloudformDesign.com. Together we can make the usertools library fantastic!

     For many years Morse code was a large part of the mandatory licensing examination in both Canada and the US. In order to get a basic license to transmit on amateur bands, a prospective amateur would have to commit him or herself to learning Morse code. This was likely a large reason why many interested in amateur radio failed to ever complete or even take their exam.

      

      In late 2006, the Federal Communications Commission dropped the Morse code requirement from amateur radio examinations. This followed a similar decision made in 2004 by Industry Canada, the amateur radio regulatory body in Canada. This move has proved to be very popular; new hams now only need to study the actual regulations of amateur radio, as well as how antennas, radios and electronics work at a fundamental level. As a direct result of this change, the numbers of new amateurs registered each year has grown at an incredible rate.

 

     Though most amateurs at the time were in favour of a relaxed code requirement, there was a substantial minority that opposed such a move, claiming it would speed the demise of Morse code usage. Whether this has happened is debatable. In both Canada and the US there is still a separate license for CW (continuous wave, which is what amateurs call Morse code transmissions) which an amateur needs to hold before being allowed to transmit on the CW bands. CW transmissions on non-CW bands are prohibited. Thus, to be able to practice CW by actually transmitting using a key is prohibited until after the amateur has demonstrated 12 words per minute proficiency.

    

J38TelegraphKey.jpg

     Many new amateurs are taking a renewed interest in CW. Because it is not a requirement that they learn it to get on the air, hams can now learn at a slower pace and also listen to CW being transmitted. There are numerous software programs that assist in teaching the transcription of Morse code. One popular method is the Koch method. The Koch method prescribes that you start with only two letters. These two letters are ‘sent’ randomly, and the listener must write them down. After a 5-minute study period, the student compares what they wrote down with what was actually sent. If the listener got at least 90% of the letters correct, they add another letter. This method has proven to work very well. The other leading method is the Farnsworth method, in which students must learn all letters simultaneously!

     My only question was ‘How can I practice sending Morse code if I can’t do it on air, and all the software I have currently is focused on the student receiving Morse transmissions?’ Well, like so many other problems I’ve had, I decided to try to solve it myself. I’ve been prototyping different methods of emulating Morse keys. The first type is the straight key, seen at left, which is what most people associate with a telegraph – a simple button on a lever that turns a switch on or off. Holding the switch down turns the switch on, creating a tone that is transmitted. Sending the dots and dashes which make up Morse code is accomplished by pushing the key down for different lengths of time. Simple, right?

 

Vibroplex_edit.jpg

     Well, many hams found that they hit a wall in terms of speed with this type of key. So the Vibroplex key was invented – a precursor to the iambic paddles that are popular today. With the Vibroplex, seen at the right, the key was turned from a vertical movement to a horizontal one. Pushing the key to the right started a mechanical pendulum-type movement that created a series of dots. Pushing it to the left simply acted as a switch, in order to create dashes. This type of key was very popular; with a little bit of practice, somebody already familiar with CW could increase their speed very quickly. The speed at which the dots are created is a function of how hard the key is pushed.

     An even faster method was found. As mentioned before, this type of key is called an iambic paddle, or iambic key. As opposed to the Vibroplex, which only automates the dots, an iambic key automates both dots and dashes. Pushing the key to one side produces repeated dots; pushing it to the other produces repeated dashes. The speed is usually adjusted either on the radio, or on the key itself. Most modern iambic keys are simply triggering the radio to create the beeps. Though they look very complex, at their most basic design level they are just two opposing switches that are actuated by levers. The paddles are engineered to be heavy so the key doesn’t slide around; they are also made to be comfortable to use for extended periods of time.

pcw.jpg

   So my next decision was which type of key to emulate. I decided to make prototypes of both straight keys (which are still popular today) and iambic keys. You can see what an iambic key looks like to the left. I'll show you how I designed both of them in this blog. The straight key was the simpler of the two, both in software and hardware. Essentially, all that is needed is a single button that causes the software to create a beep which is sent to an external speaker. However, there are always a few different ways to do the same thing. In order to get a prototype up and running quickly, I decided to use my Arduino. I have a SparkFun RedBoard, but they are functionally identical to the Arduino UnoArduino Uno. The reason I ended up with a RedBoard is that I wanted to build mine as a kit. The drawback is that the USB port isn’t integrated; I’d recommend picking up a pre-assembled Uno,  like this one  like this one. You can use any Arduino for this project, as long as you know which pins produce pulse-width modulation signals (PWM).

     In order to make this as easy as possible, I took a piece of protoboard and cut it to fit over the footprint of the Arduino headers (see below, with the protoboard upside down beside the Arduino). I then simply soldered pin headers to each side of this piece of protoboard and then attached a button, a 1k pull-up resistor, and also a couple of female header pins for the external speaker to attach to. The button is attached to digital pin 2, and the output of the speaker comes from digital pin 6, which is one of the PWM outputs.

While you could simply put a routine inside the loop() function that constantly checks to see if the button is pressed, I found it much easier to use interrupts. If you’re not familiar with interrupts, or you’ve never used them before, there are many tutorials online describing how they work.

 

     Instead of checking the button in an infinite loop (which would work, but then the processor would just be sitting there screaming along, checking the button over and over as fast as it possibly can – future expansion of the code would be difficult, if not impossible) I decided to use interrupts. When the button is pressed, it triggers what is called an ‘external interrupt’, which means the interrupt is generated from an external source. In our case this is our ‘key’ button. Understanding interrupts is simple – they are exactly what they are named – interrupts! When an interrupt occurs, the processor is interrupted in whatever it’s currently doing. The processor stores whatever it is working on in a temporary area, then immediately jumps to execute the interrupt

arduino_with_board_unmounted2.jpg

handler. The ATmega328PATmega328P has many different interrupt sources, many of them inside software. For example, you could have an external interrupt from a button. When you push the button, the processor stops whatever it’s doing, and then light a LED. It then goes back to what it was doing before. Interrupts are an extremely powerful tool. Not only do they help reduce the use of unnecessary loops, they are also vital to time-sensitive tasks like sampling an analog-to-digital converter. Because I used interrupts, I had to blend C with Arduino code. Those familiar with Arduino will see that the setup() and loop() functions are still there. I’ve just added on a couple extra functions to handle the interrupts and also what to do on start-up.

 

     Enough about interrupts – let’s move on to the beeping! The Arduino libraries include a way to activate PWM without needing to know much about the AVR registers. You’re probably already familiar with the command if you’ve used an Arduino before – analogWrite(). The name is slightly misleading, as you’re not actually writing an analog value to a pin. The analog value is being averaged by using PWM. However, in most cases – such as lighting an LED, driving the speed of a motor, etc. – the user can’t tell the difference because the PWM is happening so fast it just seems like an analog value.

PWM can also be used to create tones out of a speaker – after all, we’re just generating a frequency. If we bring that frequency down into the human hearing range, we could make the chip play Bohemian Rhapsody if we wanted! The problem is, the Arduino command analogWrite() doesn’t have any way of controlling the frequency of the PWM. So I had to use C for that part of the code. What I’m actually doing is activating the timer module attached to pin 6. I’m then telling it I want PWM mode. The PWM frequency is based off of the system clock frequency. Most Arduinos run at 16MHz, or 16,000,000 Hz. Since the timer is 8-bits wide, it can only count up to 255. So if we had the clock running at full speed, the timer would overflow once every 16,000,000/255 seconds, or 62,745 times a second. With a duty cycle of 50%, this would mean a frequency of 62.745 kHz, which is WAY too high for the human ear to listen to. This is a good frequency for motors and LEDs, but we need something around 1 kHz if possible. Luckily, the AVR designers were thinking, and included a prescaler option. This means we can divide the clock frequency by a set value. Our options are 8, 64, 256 and 1024. These would produce frequencies of 7.8 kHz, 980 Hz, 245 Hz, and 61.2 Hz, respectively. Wait a minute – 980Hz, that’s almost 1 kHz exactly! How perfect! So I chose a prescaler value of 64. This will create a tone that is very close to what most radios produce when the Morse key is pushed.

 

arduino_with_board_mounted2.jpg     Now that we have the tone out of the way, we’re essentially done! The code activates the external interrupt associated with digital pin 2, where our button is located, and when the button is pushed the interrupt turns the PWM on. When the button is released, it turns the PWM off. Wow! That was easier than I thought it would be. I threw in a start-up routine for kicks. When the Arduino is reset or turned on, it will spell out CQ in Morse code. This is the Morse code for “calling anyone”.

     I also wanted to turn things up a notch and emulate an iambic key. For this, we’re going to leave the Arduino behind and work entirely in C. This might be possible to implement in the Arduino IDE, but I find it much easier to program more complex things like this in Atmel Studio. They also have an Arduino compiler built-in, making it very easy to transition between the two programs.

     Please check out the videos below -- you might even catch me sending a bit of morse code! I'm actually quite proud of how well this tool has helped me to send Morse. I've been using a piece of software by an amateur radio enthusiast whose call letters are G4FON. He developed a piece of software for the Koch method, which you can find here. After trying out many different pieces of software, I've found that this is one of the better ones.

 

     My speed and number of letters is slowly increasing. I actually find that I can send better than I can receive! If you want to build this project, and you need help, or have questions, please contact me! You can send me a message here on element14, you can e-mail me at maxrowsell@gmail.com, or you can try and catch me on the air sometime -- VA3XMR. And also, read through my comments in the code -- I tried to explain things without making them too long.





 

Filter Blog

By date: By tag: