When running a race the homestretch refers to an area between the last turn and the winning post. It can also refer to the last part of a product design project or a new production release campaign. In each case, participants within the homestretch are often challenged to dig deep within themselves to overcome mental and physical barriers that stand between them and the finish line.
In this series of articles, we take a closer look at a few of the most frequently overlooked areas of new design starts. During the product design cycle if you overlook areas such as MAC addressing, product serialization, or factory test the first product builds can get tripped up or plagued with last minute scrambles to put measures in place to handle them or risk an impact upon your time to market. You might be wondering at this point: “I am a design engineer, why does any of this matter?” These subject areas matter because they are potential hurdles that can slip initial production schedules by days or even weeks if they are not thought about ahead of time. If we look at the graph shown below we see an idealized production yield curve.
By following the blue line we can see how much product quantity our manufacturing facility will produce over the lifetime of our product before the final build is made. Our product’s profitability is bounded by the area under the curve and our product obsolescence date is fixed, then it follows that the business folks will want to cause the ramp-up period to point A to occur as soon as possible so that the maximum amount of market share can be captured by having our product on the shelf to sell to waiting customers with money in hand. By implementing a small amount of planning upfront, you can help avoid these last minute hurdles in the homestretch to production and help reach point A as quickly as possible.
This post is the first in a short series intended to help others avoid common pitfalls encountered when releasing new products to the assembly line. On the first part of this series, I wanted to spend some time on a topic that is very near and dear to my heart as it is one of the first problems that I worked on as a design engineer coming out of school, assigning products unique MAC addresses!
What is a MAC Address?
For anyone not familiar with MAC addresses or why they are so important, every device that communicates over an IEEE 802 compliant medium must have a unique MAC address*. The IEEE maintains governance over the rules for issuance of MAC addresses and it generally breaks down to this:
When an organization wants to make new devices that will be connected up to a network they purchase a MAC address block where the IEEE assigns on OUI or an Oranizationally Unique Identifier to that entity. This OUI consists of 3 octets that identify any device produced by that organization as having originated from that OEM or one of their subsidiaries, here you can see the OUI for Avnet is shown as the 00:02:B5 set of octets. The second set of 3 octets are defined as the extension identifier and although the IEEE does not require that you assign these in sequential order, it is a strongly recommended practice since the IEEE requires application for certification that 95% of the assigned address space is consumed before being allowed to purchase another block of MAC addresses under a new OUI.
Luckily for us, development platforms that are used in engineering labs or at home for hobbyist purposes traditionally haven’t had a need for real MAC addresses unless connected up to the Internet. Now that we are in the middle of the IoT growth era, there is a clear need to have a strategy in place for assigning unique MAC addresses to all products including SOMs that are purchased from suppliers such as Avnet. In this manner, every end product can be properly identified and communicate correctly within the Internet of Things.
MAC Address Programming Approaches
Once you have obtained a block of MAC addresses from the IEEE, how can you get one of those addresses into each of your devices?
The most straightforward way to get a MAC address into an embedded device is to program the MAC address manually through a serial terminal at the factory down into a non-volatile memory so that it can be loaded up by the application at run time. This has some pretty obvious perils since it involves a human handling the product during an additional step where they could possibly enter an erroneous or, even worse, a duplicate MAC address.
So if we follow the famous Toyota practice of Jidoka (pronounced Yi-do-kah) for process improvement and defect removal, we would find that MAC addresses will become cheaper the closer you add them to the initial product assembly stages and also the quality of the product will increase as a result of reduced defects because defects like missing MAC addresses are caught early on before they reach the end of the production line. So the next logical step to reduce human involvement is to work with your IT department to come up with a way to programmatically assign your MAC addresses sequentially to each of the devices as they come off the assembly line.
The third approach we will cover is the pre-programmed component option which is arguably the best option for small to medium production volumes because of the advantage gained by low device costs, minimal component footprints, and the supplier pretty much does most of the work for you.
One fourth and final option not shown here, should not even be on the table for discussion, and that is to have a technician in the field program the MAC address which is the most expensive and risky solution. Next, let’s take a look at a quick cost breakdown for these approaches.
Approach A: Manual Programming
Unless you are building a handful of units, this is the least attractive of the three strategies mentioned. It is going to take about 1.5 minutes worth of technician time to boot up a system, connect through a maintenance UART into a terminal, run some sort of MAC address programming application, hand type the MAC address, double check the MAC address is correct, save to non-volatile memory, and then cross that MAC address off the list of available IDs. Nowadays this can be done relatively easily using U-Boot or a simple bare metal programming application so the cost of development is very low. However, the per unit labor cost can be high and a manual process is very prone to error plus there is a high potential for issuing duplicate MAC addresses.
If you are wondering how bad it would be to issue a duplicate MAC address on a device and send it out to the field, think about how many problems you would encounter if you had the same exact mailing address as someone else who didn’t live close to you. Duplicate MAC addresses are not the same exact problem, but you can easily imagine the chaos arising from getting someone else’s mail and them getting your mail. In short, one of those mis-programmed devices is either going to go in the garbage or get returned to the factory for a replacement, which is costly for business to absorb.
Let’s estimate the unit cost at 1.5 minutes for setup and programming time and multiply that times the average charged technician time rate at most CMs and we get to a number somewhere around 2.25 USD.
Approach B: Automated Programming
As I mentioned earlier, I used to be tightly integrated into this type of process at my first engineering job. The first company-wide process I was responsible for was to make sure that all of our contract manufacturers (CMs) around the world were adequately fed MAC address sub-blocks, every networked endpoint device automatically was programmed with a MAC address, and also guarantee that each networked device was given a unique MAC address. This part of my job was about 0.5 man-years of work total, but once it was in place, it required no manual intervention since the system was completely automated.
Consider this as a rough approximation of cost of doing MAC addresses yourself instead of buying a MAC EEPROM. Take a Software Engineering salary for a half of a year of development/test/support: 45,000 USD and divide that by the total number of networked units benefiting from MAC and built over 5 years: 25,000 units. Ignoring any equipment that the IT department or CM might bill to your engineering department, the amortized cost per unit is about 1.80 USD which goes down as you continue to build more volume of product. Not cheap for lower volumes, but definitely less costly and much more reliable than the manual approach once you get past any roadblocks the IT department or CM throws up for you.
Approach C: Pre-programmed Components
The best option for most manufacturing operations, unless you are talking extremely high volume, is to use pre-programmed parts. You can source specialized EEPROMs that have MAC-48, EUI-48, and EUI-64 identifiers, the identifiers are guaranteed to be unique, and the identifiers are pre-loaded at the silicon supplier for a fraction of what you might pay to do this manually.
This approach requires very little application development and the best part is that you don’t have to get your IT department involved for this! Since there is no human intervening, the reliability of having unique MAC addresses on all your product is extremely high.
If you buy these components in quantity, of course the price gets a little bit better. According to the Avnet/Newark/Farnell websites today 100+ piece pricing for is just under 0.35 USD so it is still a very good deal compared to the other approaches above. Multiplying that times the cost adder to place a part on the PCB at your CM and you are still looking at the per-unit cost being less than 0.50 USD which is pretty good for most low-medium volume product runs.
Below is a picture of the inside of the Avnet Programming Center here in Chandler, Arizona where we have the ability to provide custom programming services, including serialized ID numbers, for non-volatile memory. It is the largest programming center in North America and Avnet has similar programming centers from around the world that your customers can take advantage of for programming MAC addresses or other serialized information into memory devices.
So where is this all going? Well I have gotten questions from product engineers and designers over the years about how to read the MAC ID from an EEPROM and then utilize that MAC address from the software that they are running on their system. Thanks to a lot of help from my Avnet team, I am proud to announce that we have come up with an example design which shows design engineers how to read the EUI-48 from a pre-programmed EEPROM and configure the eth0 networking interface under Linux with the EUI as the unique MAC address. This can be demonstrated on our PCIe development platform using the “UltraZed-EG IO Carrier Card - PetaLinux 2017.4 Enhanced BSP” found on this page of our community website:
The PetaLinux BSP uses a startup script to scrape the MAC address out of EEPROM U6 on the PCIe Carrier and then call the ifconfig command to set the Ethernet interface HWADDR to match the MAC address read from the EEPROM. This is an example of how to take advantage of a pre-programmed MAC EEPROM device using approach C from above. A similar technique could also be used to read back the MAC address if the generic U8 EEPROM on the SOM were configured with a unique MAC address at build time using either approach A or B from above.
Similarly, the Avnet Starter Kit (AES-ZU7EV-1-SK-G) is an even more advanced Xilinx Zynq MPSoC development platform also uses this same approach of using these same pre-programmed MAC EEPROM devices.
Stay tuned for some additional topics that come up when manufacturing electronic systems. I also would love to hear about any topics that you have found challenging along with workarounds you came up with to get you past the last few hurdles in manufacturing a new product.