Introduction

 

Law enforcement officers are often searching for vehicles that have been reported stolen, are suspected of being involved in criminal or terrorist activities, are owned by persons who are wanted by authorities, have failed to pay parking violations or maintain current vehicle license registration or insurance, or any of a number of other legitimate reasons. Victims and witnesses are frequently able to provide police with a description of a suspect vehicle, including in some cases a full or partial reading of their license plate number. Depending on the seriousness of the incident, officers may receive a list of vehicles of interest to their agency at the beginning of their shift, or receive radio alerts throughout the day, providing vehicle descriptions and plate numbers including stolen vehicles, vehicles registered or associated with wanted individuals or persons of interest, vehicles attached to an AMBER alert, missing persons alert, and Be On the LookOut - or BOLO - alerts. These lists can be sizable depending on the jurisdiction, population size, and criteria for the list, and can present challenges for the patrol officer.

 

Automated License Plate Readers (ALPRs) function to automatically capture an image of the vehicle’s license plate, transform that image into alphanumeric characters using optical character recognition or similar software, compare the plate number acquired to one or more databases of vehicles of interest to law enforcement and other agencies, and to alert the officer when a vehicle of interest has been observed. The automated capture, analysis, and comparison of vehicle license plates typically occur within seconds, alerting the officer almost immediately when a wanted plate is observed. It automates a tedious, distracting, and manual process that officers regularly complete in their daily operations of searching for wanted vehicles. ALPR systems vastly improve the efficiency and effectiveness of officers in identifying vehicles of interest among the hundreds or thousands they observe during a routine patrol. In doing so, ALPR can identify that needle in a haystack -- the stolen car, the vehicle wanted in connection with a robbery or child abduction, or the vehicle registered to a missing person.

 

The information collected can be used by police to find out where a plate has been in the past, to determine whether a vehicle was at the scene of a crime, to identify travel patterns, and even to discover vehicles that may be associated with each other.

Automated License Plate Recognition has many uses including:

  • Recovering stolen cars.
  • Identifying drivers with an open warrant for arrest.
  • Catching speeders.
  • Determining what cars do and do not belong in a parking garage.
  • Expediting parking by eliminating the need for human confirmation of parking passes.

Watch a short video on ALPR here:

[Source: https://www.youtube.com/watch?v=LnovknVA2cE]

 

Bill of Materials

 

Items
Raspberry Pi 3B+
Rpi camera
BeagleBone AI
USB Camera

 

Raspberry Pi Vs BeagleBone AI

 

rpi_bbai

 

Raspberry Pi 3B+

[Image Source: Aliexpress]

  • Broadcom BCM2837B0 64-bit ARM Cortex-A53 Quad Core Processor SoC running @ 1.4GHz
  • 1GB RAM LPDDR2 SDRAM
  • 4x USB2.0 Ports with up to 1.2A output
  • Extended 40-pin GPIO Header
  • Video/Audio Out via 4-pole 3.5mm connector, HDMI, CSI camera, or Raw LCD (DSI)
  • Storage: MicroSD
  • Gigabit Ethernet over USB 2.0 (maximum throughput 300Mbps)
  • 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac wireless LAN, Bluetooth 4.2, BLE
  • H.264, MPEG-4 decode (1080p30); H.264 encode (1080p30); OpenGL ES 1.1, 2.0 graphics
  • Low-Level Peripherals:
    • 27x GPIO
    • UART
    • I2C bus
    • SPI bus with two chip selects
    • +3.3V
    • +5V
    • Ground
  • Power Requirements: 5V @ 2.5A via a micro USB power source
  • Supports Raspbian, Windows 10 IoT Core, OpenELEC, OSMC, Pidora, Arch Linux, RISC OS, and More!
  • 85mm x 56mm x 17mm

Beaglebone AI

[Image Source: Element14]

  • Processor: Texas Instruments Sitara AM5729
  • Dual Arm® Cortex®-A15 microprocessor subsystem
  • 2 C66x floating-point VLIW DSPs
  • 2.5MB of on-chip L3 RAM
  • 2x dual Arm® Cortex®-M4 co-processors
  • 4x Embedded Vision Engines (EVEs)
  • 2x dual-core Programmable Real-Time Unit and Industrial Communication SubSystem (PRU-ICSS)
  • 2D-graphics accelerator (BB2D) subsystem
  • Dual-core PowerVR® SGX544™ 3D GPU
  • IVA-HD subsystem
  • BeagleBone Black mechanical and header compatibility
  • 1GB RAM and 16GB on-board eMMC flash with high-speed interface
  • USB Type-C for power and superspeed dual-role controller; and USB type-A host
  • Gigabit Ethernet, 2.4/5GHz WiFi, and Bluetooth
  • micro HDMI
  • Zero-download out-of-box software experience with Debian GNU/Linux

ALPR Working

 

We are going to build a compact ALPR using Python and OpenCV. This project is inspired by from the works of Chris Dahms.

You can watch the whole steps in detail in his video here :

[Source: https://www.youtube.com/watch?v=fJcl6Gw1D8k ]

 

The code has been modified for our requirements and tested on both RPi 3B+ and Beaglebone AI.

After the license plate is recognized, the data is written to a CSV file using Panda dataset.

 

The procedure for ALPR will be briefly explained as follows:

 

1. The Image is read by the program.  For testing, let's use a random vehicle image where the license plate is visible in the frame.

bmw

[Image Source: BMW Blog, https://www.bmwblog.com/2015/12/19/why-do-americans-put-european-license-plates-on-their-cars/ ]

 

imgOriginal  = cv2.imread("test.png")

 

2. Conversion to grayscale

threshold

height, width, numChannels = imgOriginal.shape
imgHSV = np.zeros((height, width, 3), np.uint8)
imgHSV = cv2.cvtColor(imgOriginal, cv2.COLOR_BGR2HSV)
imgHue, imgSaturation, imgGrayscale = cv2.split(imgHSV)

 

3. Threshold the grayscale image

height, width = imgGrayscale.shape
imgBlurred = np.zeros((height, width, 1), np.uint8)
imgBlurred = cv2.GaussianBlur(imgMaxContrastGrayscale, GAUSSIAN_SMOOTH_FILTER_SIZE, 0)
imgThresh = cv2.adaptiveThreshold(imgBlurred, 255.0, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, ADAPTIVE_THRESH_BLOCK_SIZE, ADAPTIVE_THRESH_WEIGHT)

 

4. Contour detection

imgThreshCopy = imgThresh.copy()
imgContours, contours, npaHierarchy = cv2.findContours(imgThreshCopy, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)   # find all contours
height, width = imgThresh.shape
imgContours = np.zeros((height, width, 3), np.uint8)

 

5. Detecting contours with possible characters

class PossibleChar:
    def __init__(self, _contour):
        self.contour = _contour
        self.boundingRect = cv2.boundingRect(self.contour)
        [intX, intY, intWidth, intHeight] = self.boundingRect
        self.intBoundingRectX = intX
        self.intBoundingRectY = intY
        self.intBoundingRectWidth = intWidth
        self.intBoundingRectHeight = intHeight
        self.intBoundingRectArea = self.intBoundingRectWidth * self.intBoundingRectHeight
        self.intCenterX = (self.intBoundingRectX + self.intBoundingRectX + self.intBoundingRectWidth) / 2
        self.intCenterY = (self.intBoundingRectY + self.intBoundingRectY + self.intBoundingRectHeight) / 2
        self.fltDiagonalSize = math.sqrt((self.intBoundingRectWidth ** 2) + (self.intBoundingRectHeight ** 2))
        self.fltAspectRatio = float(self.intBoundingRectWidth) / float(self.intBoundingRectHeight)

listOfPossibleChars = []

for i in range(0, len(contours)):                 
        cv2.drawContours(imgContours, contours, i, Main.SCALAR_WHITE)
        possibleChar = PossibleChar.PossibleChar(contours[i])

        if (possibleChar.intBoundingRectArea > MIN_PIXEL_AREA and
        possibleChar.intBoundingRectWidth > MIN_PIXEL_WIDTH and possibleChar.intBoundingRectHeight > MIN_PIXEL_HEIGHT and
        MIN_ASPECT_RATIO < possibleChar.fltAspectRatio and possibleChar.fltAspectRatio < MAX_ASPECT_RATIO):
            intCountOfPossibleChars = intCountOfPossibleChars + 1           # increment count of possible chars
            listOfPossibleChars.append(possibleChar)

imgContours = np.zeros((height, width, 3), np.uint8)
contours = []
for possibleChar in listOfPossibleCharsInScene:
     contours.append(possibleChar.contour)
cv2.drawContours(imgContours, contours, -1, Main.SCALAR_WHITE)

 

6. Regrouping with matching characters

def findListOfMatchingChars(possibleChar, listOfChars):
    listOfMatchingChars = []   
    for possibleMatchingChar in listOfChars:       
        if possibleMatchingChar == possibleChar:  
            continue                              
        fltDistanceBetweenChars = distanceBetweenChars(possibleChar, possibleMatchingChar)
        fltAngleBetweenChars = angleBetweenChars(possibleChar, possibleMatchingChar)
        fltChangeInArea = float(abs(possibleMatchingChar.intBoundingRectArea - possibleChar.intBoundingRectArea)) / float(possibleChar.intBoundingRectArea)
        fltChangeInWidth = float(abs(possibleMatchingChar.intBoundingRectWidth - possibleChar.intBoundingRectWidth)) / float(possibleChar.intBoundingRectWidth)
        fltChangeInHeight = float(abs(possibleMatchingChar.intBoundingRectHeight - possibleChar.intBoundingRectHeight)) / float(possibleChar.intBoundingRectHeight)

        if (fltDistanceBetweenChars < (possibleChar.fltDiagonalSize * MAX_DIAG_SIZE_MULTIPLE_AWAY) and
            fltAngleBetweenChars < MAX_ANGLE_BETWEEN_CHARS and
            fltChangeInArea < MAX_CHANGE_IN_AREA and
            fltChangeInWidth < MAX_CHANGE_IN_WIDTH and
            fltChangeInHeight < MAX_CHANGE_IN_HEIGHT):
            listOfMatchingChars.append(possibleMatchingChar)

    return listOfMatchingChars

def distanceBetweenChars(firstChar, secondChar):
    intX = abs(firstChar.intCenterX - secondChar.intCenterX)
    intY = abs(firstChar.intCenterY - secondChar.intCenterY)
    return math.sqrt((intX ** 2) + (intY ** 2))

def angleBetweenChars(firstChar, secondChar):
    fltAdj = float(abs(firstChar.intCenterX - secondChar.intCenterX))
    fltOpp = float(abs(firstChar.intCenterY - secondChar.intCenterY))
    if fltAdj != 0.0:                           
        fltAngleInRad = math.atan(fltOpp / fltAdj)      
    else:
        fltAngleInRad = 1.5708                         
    fltAngleInDeg = fltAngleInRad * (180.0 / math.pi) 
    return fltAngleInDeg

listOfListsOfMatchingChars = []
                  
for possibleChar in listOfPossibleChars:                        
    listOfMatchingChars = findListOfMatchingChars(possibleChar, listOfPossibleChars)       
    listOfMatchingChars.append(possibleChar)               
    if len(listOfMatchingChars) < MIN_NUMBER_OF_MATCHING_CHARS:    
        continue                                                                           
    listOfListsOfMatchingChars.append(listOfMatchingChars)      
    listOfPossibleCharsWithCurrentMatchesRemoved = []
    listOfPossibleCharsWithCurrentMatchesRemoved = list(set(listOfPossibleChars) - set(listOfMatchingChars))
    recursiveListOfListsOfMatchingChars = findListOfListsOfMatchingChars(listOfPossibleCharsWithCurrentMatchesRemoved)
    for recursiveListOfMatchingChars in recursiveListOfListsOfMatchingChars:       
        listOfListsOfMatchingChars.append(recursiveListOfMatchingChars)
return listOfListsOfMatchingChars

 

7. Detection of a potential license plates

imgContours = np.zeros((height, width, 3), np.uint8)
    for listOfMatchingChars in listOfListsOfMatchingCharsInScene:
        intRandomBlue = random.randint(0, 255)
        intRandomGreen = random.randint(0, 255)
        intRandomRed = random.randint(0, 255)
        contours = []

        for matchingChar in listOfMatchingChars:
            contours.append(matchingChar.contour)

        cv2.drawContours(imgContours, contours, -1, (intRandomBlue, intRandomGreen, intRandomRed))

listOfPossiblePlates = []
for listOfMatchingChars in listOfListsOfMatchingCharsInScene:      
    possiblePlate = extractPlate(imgOriginalScene, listOfMatchingChars) 
    if possiblePlate.imgPlate is not None:                        
        listOfPossiblePlates.append(possiblePlate)  

for i in range(0, len(listOfPossiblePlates)):
    p2fRectPoints = cv2.boxPoints(listOfPossiblePlates[i].rrLocationOfPlateInScene)
    cv2.line(imgContours, tuple(p2fRectPoints[0]), tuple(p2fRectPoints[1]), Main.SCALAR_RED, 2)
    cv2.line(imgContours, tuple(p2fRectPoints[1]), tuple(p2fRectPoints[2]), Main.SCALAR_RED, 2)
    cv2.line(imgContours, tuple(p2fRectPoints[2]), tuple(p2fRectPoints[3]), Main.SCALAR_RED, 2)
    cv2.line(imgContours, tuple(p2fRectPoints[3]), tuple(p2fRectPoints[0]), Main.SCALAR_RED, 2)
    print("possible plate " + str(i)")
    cv2.imshow("Plates", listOfPossiblePlates[i].imgPlate)

 

8. Applying character recognition on the detected plate using tesseract API

listOfPossiblePlates = DetectChars.detectCharsInPlates(listOfPossiblePlates)

def detectCharsInPlates(listOfPossiblePlates):
    intPlateCounter = 0
    imgContours = None
    contours = []

    if len(listOfPossiblePlates) == 0:          
        return listOfPossiblePlates            
    for possiblePlate in listOfPossiblePlates:      
        possiblePlate.imgThresh = cv2.resize(possiblePlate.imgThresh, (0, 0), fx = 1.6, fy = 1.6)
        thresholdValue, possiblePlate.imgThresh = cv2.threshold(possiblePlate.imgThresh, 0.0, 255.0, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

        listOfPossibleCharsInPlate = findPossibleCharsInPlate(possiblePlate.imgGrayscale, possiblePlate.imgThresh)

        listOfListsOfMatchingCharsInPlate = findListOfListsOfMatchingChars(listOfPossibleCharsInPlate)

        if (len(listOfListsOfMatchingCharsInPlate) == 0):
            possiblePlate.strChars = ""
            continue

        for i in range(0, len(listOfListsOfMatchingCharsInPlate)):                             
            listOfListsOfMatchingCharsInPlate[i].sort(key = lambda matchingChar: matchingChar.intCenterX)       
            listOfListsOfMatchingCharsInPlate[i] = removeInnerOverlappingChars(listOfListsOfMatchingCharsInPlate[i])             

        intLenOfLongestListOfChars = 0
        intIndexOfLongestListOfChars = 0

        for i in range(0, len(listOfListsOfMatchingCharsInPlate)):
            if len(listOfListsOfMatchingCharsInPlate[i]) > intLenOfLongestListOfChars:
                intLenOfLongestListOfChars = len(listOfListsOfMatchingCharsInPlate[i])
                intIndexOfLongestListOfChars = i

        longestListOfMatchingCharsInPlate = listOfListsOfMatchingCharsInPlate[intIndexOfLongestListOfChars]

        possiblePlate.strChars = recognizeCharsInPlate(possiblePlate.imgThresh, longestListOfMatchingCharsInPlate)

    return listOfPossiblePlates

def findPossibleCharsInPlate(imgGrayscale, imgThresh):
    listOfPossibleChars = []                        
    contours = []
    imgThreshCopy = imgThresh.copy()
    imgContours, contours, npaHierarchy = cv2.findContours(imgThreshCopy, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

    for contour in contours:                
        possibleChar = PossibleChar.PossibleChar(contour)

        if checkIfPossibleChar(possibleChar):              
            listOfPossibleChars.append(possibleChar)     

    return listOfPossibleChars

def checkIfPossibleChar(possibleChar):
    if (possibleChar.intBoundingRectArea > MIN_PIXEL_AREA and
        possibleChar.intBoundingRectWidth > MIN_PIXEL_WIDTH and possibleChar.intBoundingRectHeight > MIN_PIXEL_HEIGHT and
        MIN_ASPECT_RATIO < possibleChar.fltAspectRatio and possibleChar.fltAspectRatio < MAX_ASPECT_RATIO):
        return True
    else:
        return False

def findListOfListsOfMatchingChars(listOfPossibleChars):
    listOfListsOfMatchingChars = []                  
    for possibleChar in listOfPossibleChars:                        
        listOfMatchingChars = findListOfMatchingChars(possibleChar, listOfPossibleChars)       
        listOfMatchingChars.append(possibleChar)                   

        if len(listOfMatchingChars) < MIN_NUMBER_OF_MATCHING_CHARS:    
            continue                            
        listOfListsOfMatchingChars.append(listOfMatchingChars)     
        listOfPossibleCharsWithCurrentMatchesRemoved = []
        listOfPossibleCharsWithCurrentMatchesRemoved = list(set(listOfPossibleChars) - set(listOfMatchingChars))
        recursiveListOfListsOfMatchingChars = findListOfListsOfMatchingChars(listOfPossibleCharsWithCurrentMatchesRemoved)
        for recursiveListOfMatchingChars in recursiveListOfListsOfMatchingChars:        
            listOfListsOfMatchingChars.append(recursiveListOfMatchingChars)            
        break

    return listOfListsOfMatchingChars

def findListOfMatchingChars(possibleChar, listOfChars):
    listOfMatchingChars = []                
    for possibleMatchingChar in listOfChars:                
        if possibleMatchingChar == possibleChar:    
            continue                                
        fltDistanceBetweenChars = distanceBetweenChars(possibleChar, possibleMatchingChar)
        fltAngleBetweenChars = angleBetweenChars(possibleChar, possibleMatchingChar)
        fltChangeInArea = float(abs(possibleMatchingChar.intBoundingRectArea - possibleChar.intBoundingRectArea)) / float(possibleChar.intBoundingRectArea)
        fltChangeInWidth = float(abs(possibleMatchingChar.intBoundingRectWidth - possibleChar.intBoundingRectWidth)) / float(possibleChar.intBoundingRectWidth)
        fltChangeInHeight = float(abs(possibleMatchingChar.intBoundingRectHeight - possibleChar.intBoundingRectHeight)) / float(possibleChar.intBoundingRectHeight)

        if (fltDistanceBetweenChars < (possibleChar.fltDiagonalSize * MAX_DIAG_SIZE_MULTIPLE_AWAY) and
            fltAngleBetweenChars < MAX_ANGLE_BETWEEN_CHARS and
            fltChangeInArea < MAX_CHANGE_IN_AREA and
            fltChangeInWidth < MAX_CHANGE_IN_WIDTH and
            fltChangeInHeight < MAX_CHANGE_IN_HEIGHT):
            listOfMatchingChars.append(possibleMatchingChar)  
      
    return listOfMatchingChars                  

def removeInnerOverlappingChars(listOfMatchingChars):
    listOfMatchingCharsWithInnerCharRemoved = list(listOfMatchingChars)                

    for currentChar in listOfMatchingChars:
        for otherChar in listOfMatchingChars:
            if currentChar != otherChar:        
                if distanceBetweenChars(currentChar, otherChar) < (currentChar.fltDiagonalSize * MIN_DIAG_SIZE_MULTIPLE_AWAY):                                 
                    if currentChar.intBoundingRectArea < otherChar.intBoundingRectArea:         
                        if currentChar in listOfMatchingCharsWithInnerCharRemoved:             
                            listOfMatchingCharsWithInnerCharRemoved.remove(currentChar)         
                    else:                                                                       
                        if otherChar in listOfMatchingCharsWithInnerCharRemoved:                
                            listOfMatchingCharsWithInnerCharRemoved.remove(otherChar)  

 

9. Overlay the license plate number to the original image

listOfPossiblePlates.sort(key = lambda possiblePlate: len(possiblePlate.strChars), reverse = True)
    licPlate = listOfPossiblePlates[0]
    cv2.imshow("imgPlate", licPlate.imgPlate)           
    cv2.imshow("imgThresh", licPlate.imgThresh)
    if len(licPlate.strChars) == 0:                    
        print("\nno characters were detected\n\n") 
        return                                          

    drawRedRectangleAroundPlate(imgOriginalScene, licPlate)             
    print("\nlicense plate read from image = " + licPlate.strChars + "\n")  
    print("----------------------------------------")
    writeLicensePlateCharsOnImage(imgOriginalScene, licPlate)          
    cv2.imshow("imgOriginalScene", imgOriginalScene)

def drawRedRectangleAroundPlate(imgOriginalScene, licPlate):
    p2fRectPoints = cv2.boxPoints(licPlate.rrLocationOfPlateInScene)         

    cv2.line(imgOriginalScene, tuple(p2fRectPoints[0]), tuple(p2fRectPoints[1]), SCALAR_RED, 2)         
    cv2.line(imgOriginalScene, tuple(p2fRectPoints[1]), tuple(p2fRectPoints[2]), SCALAR_RED, 2)
    cv2.line(imgOriginalScene, tuple(p2fRectPoints[2]), tuple(p2fRectPoints[3]), SCALAR_RED, 2)
    cv2.line(imgOriginalScene, tuple(p2fRectPoints[3]), tuple(p2fRectPoints[0]), SCALAR_RED, 2)

def writeLicensePlateCharsOnImage(imgOriginalScene, licPlate):
    ptCenterOfTextAreaX = 0                             
    ptCenterOfTextAreaY = 0
    ptLowerLeftTextOriginX = 0                          
    ptLowerLeftTextOriginY = 0
    sceneHeight, sceneWidth, sceneNumChannels = imgOriginalScene.shape
    plateHeight, plateWidth, plateNumChannels = licPlate.imgPlate.shape
    intFontFace = cv2.FONT_HERSHEY_SIMPLEX                      
    fltFontScale = float(plateHeight) / 30.0                    
    intFontThickness = int(round(fltFontScale * 1.5))           
    textSize, baseline = cv2.getTextSize(licPlate.strChars, intFontFace, fltFontScale, intFontThickness) 
    ( (intPlateCenterX, intPlateCenterY), (intPlateWidth, intPlateHeight), fltCorrectionAngleInDeg ) = licPlate.rrLocationOfPlateInScene
    intPlateCenterX = int(intPlateCenterX)              
    intPlateCenterY = int(intPlateCenterY)
    ptCenterOfTextAreaX = int(intPlateCenterX)        
    if intPlateCenterY < (sceneHeight * 0.75):                                                 
        ptCenterOfTextAreaY = int(round(intPlateCenterY)) + int(round(plateHeight * 1.6))      
    else:                                                                                       
        ptCenterOfTextAreaY = int(round(intPlateCenterY)) - int(round(plateHeight * 1.6))     
    textSizeWidth, textSizeHeight = textSize                
    ptLowerLeftTextOriginX = int(ptCenterOfTextAreaX - (textSizeWidth / 2))           
    ptLowerLeftTextOriginY = int(ptCenterOfTextAreaY + (textSizeHeight / 2))   
    cv2.putText(imgOriginalScene, licPlate.strChars, (ptLowerLeftTextOriginX, ptLowerLeftTextOriginY), intFontFace, fltFontScale, SCALAR_YELLOW, intFontThickness)

 

9. Write the Data to a CSV file

raw_data = {'Date': [time.asctime( time.localtime(time.time()) )], 
    'PlateNumber': [licPlate.strChars]}
df = pd.DataFrame(raw_data, columns = ['Date', 'PlateNumber'])
df.to_csv('data.csv')
print("Data written to file")

Conclusion

 

  1. The testing was initially done with Raspberry Pi 3B+ and was later extended to BeagleBone AI.
  2. Some characters are misread. Eg: The algorithm wrongly interpreted 'B' as '8' and also interpreted the left section of the plate to be '1'.

          However, this can be rectified by using a different model or by training a model according to our dataset which is a very paining task.

  1. The testing was done on a series of still images from various internet sources. Testing with realtime video source is yet to be done.

 

What Next

 

The program will be extended to relay the information in real-time to a cloud server so that it can be accessed by permitted personnel.

 

References

 

ALPR:

SBC's

Algorithm: