Meet ContextCheck: Our Open-Source Framework for LLM & RAG Testing! Check it out on Github!

in Blog

March 11, 2021

License Plate Detection Technology. How does it work?

Author:




Edwin Lisowski

CSO & Co-Founder


Reading time:




12 minutes


Today, license plate detection technology is becoming more and more popular. What started with just police and speed cameras is now a common solution that allows companies and various institutions to manage parking facilities, collect tolls, and even grant access to specific areas and buildings. From a technical standpoint, license plate recognition technology is based on two AI-related solutions: Machine learning and computer vision. Let’s take a closer look at this plate recognition technology.

The scope of usage of license plate recognition technology is extremely broad. For instance, this solution can be used in:

  • Tracking stolen vehicles
  • Managing parking facilities
  • Traffic enforcement
  • Access control
  • Automatic toll collection

Nowadays, license plate detection is used not only by the police, but also by parking facility managers, security agencies, and other traffic control institutions. In this article, we are going to take a closer look at license plate recognition and see how this technology works. For starters, let’s analyze the backstage of license plate recognition.

License plate recognition: Technology

Generally speaking, we could say that license plate recognition is a blend of two (or sometimes three) major AI-fueled technologies:

In its essence, license plate recognition is simply a type of OCR (Optical Character Recognition) application. In short, Optical Character Recognition is a technology that enables you to convert different types of documents and images, such as scanned paper documents, PDF files, or images captured by a digital camera, into editable and searchable data.

In our case, we are particularly interested in license plates, and they comprise letters and digits. The OCR software has to be capable of reading every sign that can be found on a given plate. But there’s more.

We also need machine learning. What for? You’ll find out in a minute. A cutting-edge OCR software can read and analyze various fonts, colors, different text rows and columns, and sometimes even blurred images.

license plates, license plate recognition

How does OCR work?

To explain this issue, let’s use a straightforward example–a high school diploma. You can use your scanning device to put your diploma’s image into a computer, but it’s still a typical image file (JPG, PNG, etc.), which means it’s not editable. There are just some random pixels in this picture from your computer’s perspective, but no letters and digits. That’s why you need a much more advanced graphics program to edit it. Of course, that takes time, cannot be automated, and requires some advanced skills.

With OCR, you can automate this task. You see, an OCR software can take a typical image file, analyze it, single out letters, put them into words, and then–combine words into whole sentences. As a result, everything happens almost instantly, and you don’t need to be a professional graphic to edit your scanned image. Now, the computer can read your diploma and detect where your name or your degree is. This solution allows you to access and edit the document’s content at once.

Let’s go further. The most advanced OCR systems are focused on replicating natural text recognition. How does it work? Firstly, the observed object (in our case, a license plate) must be considered one entity comprising many interrelated parts (digits and letters). Second, any data interpretation must always serve some purpose (in our case, it’s text extraction).

And finally, the OCR program has to be capable of self-learning (and this is where machine learning steps into play).

OCR (Optical Character Recognition)
Now, usually, an OCR process consists of several stages that involve different algorithms. Together, these stages ensure accurate output.

Image acquisition

This is where computer vision is indispensable. Image acquisition is all about capturing and processing images and videos. Usually, computer vision tools work based on a set of predefined rules, related to finding patterns and/or identifying objects in an image. In our case, we’re looking for vehicles and their license plates. What happens in a situation where there is more than one vehicle is a single picture? A decent OCR software should be able to identify all of the visible license plates.

When it comes to image acquisition and analysis, data companies frequently use the OpenCV library. According to their website, the library has over 2,500 optimized algorithms, including both computer vision and machine learning. Companies and single programmers can use these algorithms to:

  • Detect and recognize faces (so-called face recognition)
  • Identify objects
  • Classify human actions in videos
  • Track camera movements
  • Track moving objects
  • Extract 3D models of objects
  • Find similar images from an image database

Detect and recognize cars, road, license plate recognition
The OpenCV library can help you with several important processes related to license plate recognition:

  1. Image enhancement: Let’s take the example of noise. In most cases, not only is it useless, but it also deteriorates the image. Noise originates from the camera itself and usually happens when pictures are taken in dark places. The OpenCV library has appropriate algorithms that help with image denoising.
  2. Information extraction: The extracted text is collected from the image and transferred to the given application or a specific file type.
  3. Background removal: Within this stage, the background is removed and replaced with transparency or solid color (particularly white). As a result, you have a picture where only the object in question is visible, and nothing distracts the viewer. Admittedly, background removal is especially common in the e-commerce sector, but not necessarily in license plate recognition.

Image localization

The next step is to localize (identify) what particular areas in the specific image mean. In many instances, several aspects need to be considered, just to mention the distance from a camera, angle, vehicle’s movement, and direction in which it’s going. And what about the license plates themselves?

Here, the situation is simplified. You see, they have one shape and the same font, which makes them easy to identify.

However, there can be more than one rectangular object on a vehicle, right? That’s why you have to teach the algorithm to take into account more than just the shape and other features that help the OCR software recognize a license plate correctly.

IMAGE LOCALIZATION, map, phone, pointing finger

Image normalization

As we’ve already discussed, not every image will be of the same quality and angle. The OCR algorithm has to bring these various images to an optimal size and proportions that will facilitate the text extraction process. Let’s assume that the algorithm detects several vehicles from multiple angles at the same time. Our algorithm has to detect all the license plates in one picture and regulate the contrast and brightness. It all happens automatically.

Moreover, for obvious reasons, the car’s image in motion is usually far more complicated to process than the one of a stationary vehicle. And as you already know, the algorithm also has to deal with various weather conditions and a time of day. All of that happens in the image normalization stage.

Text extraction from image

The next step is to segment characters in the license plate. You have to know that the OCR algorithm does not analyze the license plate as a whole. No, the text and digits are divided into separate characters. The license plate number is divided into letters and numbers based on such parameters as the font’s color, the distance between characters, the font itself, and its structure.

car license plates, license plate recognition
There are many types of text extraction techniques that can be used in license plate detection. Let’s analyze five of the most popular ones:

  • Region-based method: This approach relies on several factors, such as color, edge, shape, contours, and geometry features.
  • Texture-based method: This method uses various kinds of texture and their properties to extract text from an image file.
  • Hybrid technique: It’s the mix of the previous two techniques.
  • Edge-based method: Here, you detect the edges of every sign that can be found on a license plate
  • Morphological-based method: With this approach, you extract all text-related features from the image file.

OCR with machine learning

When it comes to OCR, we have to start with teaching the computer what constitutes a text and what constitutes a license plate. In order to do so, machine learning algorithms are fed with thousands of different pictures of license plates (coming from various sources and of varying quality) so that these algorithms can understand what a license plate is and what it contains.

This task is a bit simpler when you have access to high-quality, legible pictures of license plates, where the whole text is clearly visible. However, in the real world, things are not so simple. Frequently, license plates are dirty or covered in snow. Such samples also should be used to train the ML algorithm in order to make it more effective.

If you used only high-quality photos, the algorithm would not handle the situation where the license plate is dirty or partly blurred (i.a. due to rain). That’s why you need to train your algorithm with as many different photos as possible.

Usage of the OCR software in license plate recognition

You have to understand that the OCR technology can serve lots of different purposes. We started with our high school diploma example, but these applications can be used to read the printed page from a book or a random image with text and read street signs, car license plates, and even captchas.
Typically, the OCR software takes into consideration the following factors and attributes of the image file[1]:

  • Text density. Indeed, the text density varies. For instance, on a printed page, the text is quite dense. However, on an image of a street with a single street sign, the text is sparse. The OCR software has to be able to read the text in both situations.
  • Text structure. Again, the text on a book page is usually neatly structured, while text in the wild may be scattered everywhere, in different rotations, shapes, fonts, and sizes. The OCR software has to be capable of dealing with both situations.
  • Font. And we’re not just talking about computer fonts. What about handwriting? It can be much more difficult to read!
  • Artifacts. Again, artifacts are common on the low-quality images taken in the wild. The OCR software has to recognize them and separate them from the text and digits.

handwriting

License plate recognition: Devices

When it comes to license plate recognition, we have two major components:

  1. A camera
  2. An analytics software

Now, what are these components responsible for? Let’s start with a camera. In essence, a camera captures the image of the license plate number of a specific vehicle (either stopped or moving). If the police are using such a camera, it can also help identify a suspect’s face. These cameras can be placed on a vehicle (police car), set by the road (speed camera), or in the parking lot (access control).

And then, we have the analytic software that is needed to read the license plate number. Depending on your software’s capabilities, it can help you with:

  • Security and surveillance
  • Traffic management
  • Toll collection and parking control

Let’s see how that works in real-life conditions.

street camera

Unipark – a private car with the license plate recognition system

Unipark is a company that operates in several European countries, such as Poland, Lithuania, Latvia, Estonia, and Belarus. Their car parks are equipped with advanced license plate detection systems that help them manage cars driving in and driving out:

  1. First, when a vehicle approaches the barrier that blocks access to the car park, the camera takes a picture of the license plate
  2. The image is sent to the company’s central database
  3. The barrier automatically opens
  4. You drive in, let’s say for an hour
  5. After an hour, before you can drive out, you have to go to the ticket machine, select your license plate, and pay the toll
  6. When your car approaches the barrier, the system verifies whether you paid your stoppage and, if you did, opens the barrier automatically.

Let’s think about what happened with your plate’s picture? When the image is taken, the system automatically sends it to the company’s main database. Here, everything starts with text recognition.

When that part is done, the software extracts the car’s number plate and processes it into plain, editable text, written in regular font.

private car park
The brilliance of this solution is based on the fact that everything happens automatically. There’s no need to hire parking guards, and the car owner doesn’t have to worry about a printed ticket because there is no ticket! Everything happens within the company’s central software. All you have to do is to pay for your stoppage.

Capabilities of modern cameras

As you already know, the stationary cameras can be placed at specific locations in order to record all the cars entering and leaving the particular area. For the more complex applications, you need multiple cameras placed in different locations that work at the same time. This way, for example, you can capture an accident from various perspectives. Many OCR cameras can even record a short video and, if necessary, send it to the central database.

And what about mobile cameras? Typically, they are attached to police cars and used to capture license plates, time, and date when the car is parked and where. This way, they help the police monitor various city areas. Naturally, it is vital that all these cameras can record and take pictures at any time of the day and in any weather conditions.

It’s time to sum up this article! As you can see, license plate recognition technology is quite complex and entails some serious processing work. However, license plate recognition is extremely helpful and simply necessary in today’s environment, especially in large cities. If you’d like to start using license plate recognition in your company–we are at your service! We will gladly show you what this technology entails and how it can be used to automate work in your company. Drop us a line today!

References

[1] Gidi Shperber. A gentle introduction to OCR. Oct 22, 2018. URL: https://towardsdatascience.com/a-gentle-introduction-to-ocr-ee1469a201aa. Accessed Mar 11, 2021.



Category:


Computer Vision