In this blog, I will, in short, try to explain about PYNQ- What, Why & How.
An in-depth in detail everything about the use is already described in this tutorial workshop.
PYNQ-Z2: Embedded Vision Workshop (Zero to Hero) Series with Adam Taylor (Project Deadline Extended Till July 28th)!
We generally used to do embedded system programming using different Architecture based MicroController or for advanced DSP Processor or some other ASIC.
FPGA comes in picture with its reconfigurable architecture, acceleration computing & parallel execution facilities.
SoC is a hybrid MicroController -FPGA single Integrated Chip Processor that facilitates both the pros of Microcontroller & FPGA in a single processor IC.
ZYNQ is such a specific architecture based SOC manufactured by Xilinx. PYNQ-Z2 & Ultra96 are two low-cost ZYNQ SoC based development platforms through which learning can be done & also basic prototypes can be built.
In general FPGA programmers use VHDL-Verilog languages to define hardware-level descriptions to do the programming while Software Programmers use C/C++ or Python-like high definition language to code the problem statement.
PYNQ stands for Python for ZYNQ. It provides an easy to use Python Interface through Jupyter Notebook Web Access.
Both the PYNQ-Z & Ultra96 hardware have this facility. PYNQ has Zynq®-7000 SoC while Ultra96 has Zynq UltraScale+ ™ MPSoC which results is some pros & cons. Pros are in terms of speed of operation & Cons in terms of the comparative cost difference.
PYNQ-Z boards have HDMI-in, HDMI-out ports, Arduino headers, RPI compatible 40pin GPIO headers which facilitates direct interfacing of different types of sensors & motors besides FPGA's basic plus points.
PYNQ is a device driver programming firmware operating system that is being developed using petalinux. So even if we are building a custom hardware platform using ZYNQ ICs we can build a custom PYNQ platform for the board. These boards processor mainly consists of two parts known PS(Processing System) & PL(Programming Logic). PL uses a bit file that results from programming a specific hardware configuration using Vivado- Vitis Software. Through PS we generally first call those bit file and then using python script run the program. We can create different Operation specific hardware bit files & call them for different types of problems embedded solution. This hardware approach for creating solutions provides high-speed execution being power efficient compare to only a software-based programming approach. Even we can build a state of the art deep learning-based solution's hardware PL logic and call them through python programming using this ZYNQ platform with PYNQ. This platform development environment facilitates even the software programmer to directly work on embedded solution building.
First, we have to download this PYNQ image file and burn it on the memory card using etcher. Then inserting the memory card, we have to power on the hardware platform. Next, we have to connect the hardware from the specific IP of the web interface of the hardware. This will open the jupyter platform. Then on it, we have to call the specific PL bit file known as overlay & write an application-specific python program to execute it.
For the PYNQZ2 hardware platform, we initially don't need to know about the IP connected to the board. we can directly type "pynq" in the browser and press enter to directly open the jupyter notebook interface.
Now in the right side if we click New & from dropdown to terminal it will open the new terminal window.
if you haven't logged in the board initially or by anyhow press the logout while executing, it will ask for the password. The board comes with by default username 'xilinx' & password 'xilinx' which user may change as per their interest.
Now using the "ifconfig" terminal command we can get the value of IP it's taking which we can use to access the board through putty or winscp or samba interface to modify access security settings on different folder or to transfer specific files to specific directory location.
Session 1 of the workshop covered two overlays that are included in the standard PYNQ image - the Base overlay and the Logictools overlay as well as running a simple Object Classifier CNN Model.
First Example shown is accessing the on-board button & leds.
In this example the wires connection are of no use.
Next Example shown is the execution of a simple boolean algorithm based on the implementation of XOR Gate.
We can check on different simple boolean algebra just by simply modifying the "logic operator" used and also changing nos of input.
I have tried different 2 and 3 input AND, OR, XOR, NOR, NAND and found it's successfully running crosschecking the seen output value with the truth-table results.
I haven't shared all those results here as it will unnecessarily increase the blog length & there is nothing too much new in it. Anyone can easily try & implement it.
Next useful example is the generation of specific wave-pattern and traces the output waveform.
In the below example it's only directly tracing the outputs of D17 D18 D19 which are manually connected through wires to D0 D1 D2, which are hardcoded to the specific signal patterns.
We have used WaveJSON to view both the given signals & result signals output waveform pattern.
We can create any new type of wave-pattern which not only directly takes the signal but also used some mathematic operator.
We can also create easily any type of counter circuit and trace it through the generated waveform.
We are very much familiar with the 'Finite State Machine' which is an easy to use approach for solving any type of complex problems like traffic light controller, specific string search.
'logictools.bit' hardware file has inbuilt feature of the creation of FSM. we can as all trace the output & see in waveform like the previous approach.
we can see this hardware board Arduino & Raspberry pi type pin configurations too.
Therefore we can, not only trace the output from those pins but also interface different types of motor & sensors for specific embedded applications.
This board provides a separate interface for different PMOD modules.
This board also has audio, usb-camera, hdmi-camera interfaces through which we can make realtime image processing based applications easily.
Once audio is able to capture we can create easily implement any of the audio-signal-processing algorithms.
My AUX Microphone is not working & I have no PMOD modules, so, for the time being, I have to skip those application checking.
Microblaze is a soft microprocessor core designed for Xilinx field-programmable gate arrays (FPGA) which can be used to bypass on-board ARM Processor workload in need.
The board comes with two hdmi ports, HDMI-IN and HDMI-OUT.
Through HDMI-IN we can connect hdmi camera & Through HDMI-OUT we can connect hdmi display.
In the below example, we read hardware-internally stored image and display it in the jupyter notebook as well as connected HDMI Display.
We have also extract some channel and display specifically that on our HDMI Display
[first one is color and second one is only "L" channel but its not too much clear from my phone clicked photo here]
Next we will move towards camera interfacing.
This board failed to run with GO-PRO HERO-7 HDMI camera because of the default resolution support given.
I will try to run with Logitech C270 webcam which is mostly used in most of the research-based computer vision applications.
In the below example, we have read the image directly from the webcam and do show some basic opencv based edge detected algorithm's implementation alongwith display of the output directly to the connected HDMI Display.
This confirms this board can be used for any of the OpenCV based algorithms implementations for embedded vision solution approach.
An alternative solution to test hdmi-camera based code if working or not, i found an solution though it's not handy to use for embedded soluition development.
It is use your computer screen's hdmi extension to directly connect with the HDMI-IN of the board, but if one is using to code having a single computer than it will be not too much handy solution. we can program a raspberry pi with RPI camera to act as an hdmi camera device and use it too. I am planning to make one rpi-based hdmi camera & waiting to receive the rpi+rpi cam.
Modern era is the age of mostly deep learning/ machine learning based solution rather than a classical image-processing solution through there are specific benefits over one another.
This board has facility to code it for specific deeplearning model based object classification, object detection, segmentation, pose estimation like solutions.
Below we can see examples of QNN based one object classification and one object detection neural network based deeplearning solution approach.
Some of the others neural network deeplearning model like this can be found in PYNQ Community forum website for trying and learning.
In Session 1, we have only seen importing of the overlay bit files directly and using it to modify the PL part of the hardware processor and then use PYNQ based jupyter notebook platform to program it through the PS programming.
But how are those files generated? That is created from Xilinx provided VITIS-VIVADO Software Platform.
In Session 2, we have seen how to generate a test_pattern_generator pattern specific hardware files and use it to create a pattern in jupyter notebook python script.
Below the design which is specific to the tpd_generator shared in the tutorial.
Next through jupyter notebook interface, we have write an ipynb scipt which will create a specific test pattern.
In the given folder we can see three file with bit,hwh & tcl extension those need to copied from the below-shown directory location once your created design is run completely.
These three are hardware-specific files which is needed to program the PL of the processor.
Other two files of that folder are used to take those hardware files to program the PL.
Below you can see the details of those files
and then created tpd_pynq.bit overlay file is called in jupyter notebook like shown below.
In the above part, we can modify lines value i.e 512 here & value 640 inside VideoMode i.e nos of columns.
For generating new types of test patterns, we need to refer to "Test Pattern Generator Product Manual" and modify the value under "tpg.write (0x20, ***)"
accordingly to the value shown below.
We can get the idea what these other tp.write function is calling & writing from below table from "Test Pattern Generator Product Manual".
In Session 3, we have seen the tpd_generator project have been modified to read a bar code through an hdmi camera using OCR based implementation
Here we have used a given front_end.tcl to rebuilt already build in code from some repository.
IO_Contraints.xdc need to modify as per hardware-specific.
I failed to able to run this code as the design file generated from front_end.tcl has used hdmi-camera interface and once my hdmi camera is ready i can be able to check this on hardware.
For more understanding of the details analysis on the using of vivado to create those hardware-specific files, we can go to the official pynq workshop github repository's 4th session part.
Here you can see three tcl files which are when sourced on vivado three of the separated projects are created like shown below.
once these projects are run separately in the vivado interface corresponding bit & hwh file will create which we need to copy from the specific directory as i mentioned earlier.
the block diagram of those are as below
Design specific constraints files are also needed to be added accordingly.
Once these design files are opened and analyzed anyone any easily create & add new ip block and generated new block design running it will result to generation of new hardware files.
Importing to three hardware file as shown earlier we can write a jupyter notebook python file to program the PYNQ's PL part & correspondingly program the PS as per the need.
As of the final project submission, i am going to capture a scene through webcam and segment the captured image based on basic color based segmentation and display the segmented part in the hdmi monitor connected
the overall program and basic setup are shown below which can be easily understood from only the basic things whatever I explained through the above basic examples
This is a basic too simple example code of detection specific colored object and segments it. We can change the color value limit to segment other specific zones of color as well as based on other color channels. after that we can modify the segmented part to detect the nos of objects in it or to find area of the segmented object or to find the centroid of the segmented part and use it in tracking or robotics object picking movement like different applications. we have displayed the output as color segment part here which also can be modified according to need like overlay segmented part on the original image with different color or show tracking movement or anything else.