|Product Performed to Expectations:||10|
|Specifications were sufficient to design with:||10|
|Demo Software was of good quality:||9|
|Product was easy to use:||9|
|Support materials were available:||10|
|The price to performance ratio was good:||10|
|TotalScore:||58 / 60|
Thanks to element14 and the Raspberry Pi Foundation for selecting me to do this RoadTest. It is a head to head comparison of the Raspberry Pi 4 Model B to the Pi Model 3B+ while performing face detection with OpenCV. Benchmarks released with the Raspberry Pi 4B indicate increases in speed of 2 to 4 times in many instances. Will improvements of this magnitude be realized in real world applications like face detection using OpenCV? Is a 4 GB Pi 4B required for face detection using OpenCV with a small dataset or is 1 GB enough? What is the sweet spot for training photo size? In this RoadTest these questions will be addressed and more…
UPDATE 30 August 2019: Plots of CPU and memory utilization added
UPDATE 31 August 2019: Clarified potential contribution of DDR4 RAM in Pi 4B
There have been a number of benchmarks published, including this one at element 14 and this one in the Raspberry Pi Foundation published MagPi. However, the Pi 4 is relatively new and information on the improvement gains from upgrading existing projects is limited. PiCasso Adapting Art to Viewers is a project running OpenCV running on a Rasbperry Pi Model 3B+ that I recently published on element14.
It features two Raspberry Pi computers – a 3B+ to do facial recognition and a 3B to control the video output. Two computers are necessary because of the computing demands of facial recognition. At the conclusion of the project it was noted that facial recognition frame rate was slow and would benefit from being sped up. In addition, there were a number of false identifications which would be interesting to explore further.
Below are benchmarks plucked from the Magpi that illustrate the improvements a Pi 4B might be expected to deliver relative to a Pi 3B+.
This RoadTest will be a performance comparison using a small dataset facial recognition application and OpenCV. Doubling the frame rate would be considered a significant improvement and from the table above it appears that this might be possible. The RoadTest includes:
Unboxing and Getting Started
The Raspberry Pi arrived undamaged with the retail box well packaged in a larger element14 box.
As is widely known by now the Raspberry Pi 4 has a number of new and improved hardware features:
The Raspberry Pi is not ready to use out of the box. Even those who are already using a Raspberry Pi may need to purchase the following:
In addition, for this RoadTest a keyboard, mouse, and the official Raspberry Pi Camera Module V2 were used.
Although there are known issues with the USB-C power, I did not experience them. The Raspberry Pi 4 can run hot as can previous versions. This was experienced and the solution used during the RoadTest is described below.
Of the hardware changes, the one that I found most troublesome was the dual micro-HDMI outputs. While some may find dual HDMI useful I expect most users won’t. My issue is that I use the same monitor for development on several devices and switch them frequently. The HDMI cable is difficult to unplug from the monitor due to position and I elected to use an adapter for HDMI to micro-HDMI at the Raspberry Pi. Micro-HDMI is inherently a weaker connection and after a while the adapter failed due to stresses put on it during plugging and unplugging. In addition, it is just another uncommon cable / connection to have to buy. The replacement adapter I purchased after the first failed has a short length of cable between the female HDMI and male micro-HDMI parts of the adapter and hopefully will last longer.
All in all though, the upgrades are greatly welcomed.
Operating System and Software
The Raspberry Pi 4B requires the new Rasbian image Buster, which is based on the latest version of Debian Linux. Python 3.7 is the version of Python 3.x which comes with Buster. One of my concerns was whether OpenCV would run without too much drama on the Raspberry Pi 4 with these changes.
Getting started instructions for the Raspberry Pi 4 Model B can be found here and Buster installed easily and without a problem. In addition to the normal changes in setup for keyboard and environment, be sure to enable the camera for face recognition.
Unfortunately, installing OpenCV using the method described in my previous project broke during the Build and Install. The problem took several weeks to resolve and is described here. Users of the Pi 4 should be aware that not all software and methods that ran on Debian Stretch will run without issue on Buster. And the Pi 4 will not run on Stretch.
A recipe for building OpenCV on Rasbian Buster and the Raspberry Pi 4B is given here (my thanks and gratitude to the author). Be prepared to spend several hours downloading, compiling, linking, and installing.
There are a couple of changes in the installation for the RoadTest from the one used in the PiCasso Art Deluxe Project:
Therefore it is not a direct comparison to that project but the contrasts presented here between the Raspberry Pi 4 and the Raspberry Pi 3B+ are representative of what might be expected in similar OpenCV face detection projects with small datasets.
The Pi was heavily loaded during the OpenCV build process and started to show the dreaded thermometer in the upper right hand corner indicating it was throttling back due to overheating. In this post I describe the case and fan that was designed to overcome the problem. The case is suitable for both the Raspberry Pi 4B and the Raspberry Pi 3B+. No temperature throttling was observed during the RoadTest.
NOTE: The primary purpose of this RoadTest is to determine the relative increases in speed that might be obtained when doing face detection with small datasets on a Raspberry Pi 4. The datasets have not been optimized, nor have the settings for OpenCV. In addition this is not a tutorial on face detection although there should be enough information to replicate the tests.
Both the Raspberry Pi 4B and the Raspberry Pi 3B+ ran the tests using the same microSD card and installed in the same case with the fan running to assure maximum speed.
There is a treatise on facial recognition here. The author does a great job of explaining and most of the tutorials and such for the Raspberry Pi on face recognition seem to be based on (or copies of) his work. Since there is no way I could improve on it please follow the link if interested in knowing more.
The parameters examined during the test are:
Data recorded includes time to train, approximate frame rate, encoded dataset size, Raspberry Pi memory utilization and CPU usage.
The primary training set has a total of 64 photos of seven different faces (me, the cast of Seinfeld, the Prime Minister of Denmark and Supergirl) with a resolution of 400 x 600 pixels. The number of photos of each face varies from 22 to 4. Training photos for each person are stored in their own individual folders where the encoder can find them. The training set is relatively small and evaluation with a larger dataset would be interesting but was not done.
The influence of training photo size was examined at 50x75, 100x150, 200x300, 400x600, 800x1200, and 1200x1800 pixels on a subset of photos of me, 22 in all. The full training set containing all the photos is sized 400x600.
During my previous experiments the size of the training photos was not examined and it was assumed that larger photos gave better results. However, larger photos take longer to encode and the value of larger photos if any had not been examined. A cursory review of information available on the internet revealed that there is a minimum size sometimes correlated to interpupillary distance in pixels. In the training photos used here the interpupillary distance is approximately one third of the width of the photo or a bit less. So it is varying from something around 15 pixels (one third of 50 = 17) to as much as 400 pixels (one third of 1200 = 400).
The images captured during face recognition are resized before evaluation in order to keep frame rate up with most of the tests done at 500 pixels in width. The impact of decreasing and increasing this value was evaluated by testing at 400, 500, 800, and 1200 pixels in width. Decreasing width increases frame rate but presumably might reduce recognition accuracy for subjects further from the camera. Increasing width has the opposite influence. The following screen shots show the resized frames used face recognition with training photos of 400x600 pixels.
The Task Manager was used to get a rough idea of CPU utilization and memory being used. These numbers are approximate in that they fluctuated some during the tests. As well, the OS may have had tasks running at some points that would contribute to CPU utilization and memory usage and warp the results somewhat. Python code was used to determine the time required to encode all of the training photos. Results are shown in a terminal window as the test progresses and frames per second are calculated when the test is exited.
Since the frame rate is dependent upon how many objects are identified as faces it was necessary for me to remain in the frame and relatively still at a fixed distance during testing to get consistent results. Where testing was done on faces other than mine an Apple iPad with the photo was held in front of the camera at a fixed distance. The test was stopped and the results were recorded after 30 seconds.
The video below gives an idea of how the data was captured and how it looked on the display.
The raw data for the two versions of the Raspberry Pi is given below.
Training Photo Size
The training photo size was varied from 50x75 to 1200x1800 pixels. All photos were medium resolution JPEG. There was some increase in CPU memory usage with the very largest photos that I cannot explain as shown in the plot below.
It may be that there is a cache during encoding that is not released during face detection. The primary impact of training photo size is on the encoding rate. This is illustrated in the graph below showing photos / second encoded as a function of training photo size for the Raspberry Pi 4 (blue) and the Raspberry Pi 3B+ (orange). A higher encoding rate is better.
The Raspberry Pi 4 is about twice as fast at encoding data as the Raspberry Pi 3B+. Encoding speed is highly dependent on the photo size but size of the resulting encoded data used for face detection is relatively independent of the size of the training photo. An increase in false detections seemed to be present with the very smallest training photos (50x75). Further experiment is necessary but it appears that a sweet spot in terms of encoding time and accurate face detection may occur around 100x150 to 200x300 pixels. This is in keeping with some of the information found while researching photo size. Larger photos in the range of 800x1200 and above appear to be overkill.
Training Dataset Size
The training dataset size was varied from 4 to 64 photos and evaluated. In the chart below frame recognition rate is on the X axis with the blue bars representing the Pi 4 and orange bars representing the Pi 3B+. All data is for a single face with the exception of a group detection of the Seinfeld cast containing 30 faces in the training dataset and marked on the chart. A higher frame rate is better.
For a single face varying the number of photos in the training set had little (if any) impact on frames per second during face recognition. The Pi 4 evaluated around 2.5 FPS with a 3.6 times increase in frame rate over the Pi 3B+ overall.
Of interest is the influence of multiple faces in the recognition frame. For 4 faces the frame rate dropped to 0.73 for the Raspberry Pi 4 while the Pi 3B+ dropped to 0.22 FPS. The Pi 4 was around 3.3 times faster in this case. This implies that the detection of a face and evaluating it takes more time than comparing it to the already encoded faces in the dataset (at least for the small datasets in use here).
The average encoded size of the datasets (Y axis) is largely a function of the number of photos (X axis) as shown in the nearly straight line relationship plotted below.
The encoded size averages 1.05 kB per photo with the OpenCV parameters used in this test, regardless of the size of the training photo itself.
Frame Evaluation Width
During face recognition the user can set the width of the frame being evaluated in pixels. Setting the width smaller speeds up evaluation but a larger width captures more data, which may be important for faces at a distance. The following chart shows the influence of frame size on frame rate. The Pi 4 is the blue line and the Pi 3B+ is orange. Higher frame rate is better.
At a frame width of 400 pixels the frame rate is 3 FPS for the Raspberry Pi 4 and 0.8 for the Raspberry Pi 3B+. It drops to 1.6 FPS for the Raspberry Pi 4 when frame width is 1200 pixels and 0.4 for the Raspberry Pi 3B+. Subjectively the increased frame size also contributed to increased false recognitions with a busy background. Accordingly frame width should be minimized where frame rate is important and busy backgrounds avoided.
Throughout the tests memory usage hovered around 300 MB during face recognition for both the Raspberry Pi 4 and the Pi 3B+. One exception is when the frame evaluation width is increased as shown in the plot below.
This would imply that much larger dataset could be handled even with a 1 GB Raspberry Pi. As noted by Shabaz in the comments below the faster DDR4 and extra RAM is contributing to the speed increases observed earlier but the contribution was not quantified.
Also interesting is that CPU percent utilization for the two computers was essentially the same and did not vary much from 39% on average when evaluating a single face. Again, an exception was observed with frame evaluation width.
In the plot above using single faces an increasing frame width results in increasing CPU utilization. When multiple faces were present the CPU utilization dropped - perhaps because frame rate dropped and the video screen was not being updated as frequently.
For the small dataset evaluated in this RoadTest:
The following applies to both Raspberry Pi models tested:
Problems were initially encountered getting the software to run and that was not entirely unexpected while the differences between the Raspbian Buster image and Stretch are resolved. Users should be aware of this however, especially for code that is outside that supplied by the Raspberry Pi Foundations or isn’t widely used and updated by the community.
The Raspberry Pi Model 4B delivered as promised and is a bargain. Even with the software glitches experienced and minor hardware issues it is bound to be very popular and the glitches soon resolved.