Object Tracking in Computer Vision - GeeksforGeeks (2024)

Object tracking in computer vision involves identifying and following an object or multiple objects across a series of frames in a video sequence. This technology is fundamental in various applications, including surveillance, autonomous driving, human-computer interaction, and sports analytics. In this article, we will discuss in-depth about object tracking in computer vision.

What is Object Tracking?

Object tracking, a critical component in the field of computer vision, refers to the process of identifying and following objects over time in video sequences. It plays a pivotal role in numerous applications, ranging from surveillance and traffic monitoring to augmented reality and sports analytics. The genesis of object tracking can be traced back to simpler times when algorithms were rudimentary and often struggled with basic motion detection in constrained environments.

Object Tracking vs. Object Detection

• Object tracking and object detection, while closely related in the field of computer vision, serve distinct purposes. Object detection involves identifying objects within a single frame and categorizing them into predefined classes. It’s a process of locating and classifying objects, like cars, people, or animals, within an image. This technology forms the foundation of various applications, such as facial recognition in photos or identifying objects in satellite images. Detection is a critical first step in many computer vision tasks, setting the stage for further analysis or action.
• Object tracking, on the other hand, extends beyond the identification of objects. Once an object is detected, tracking involves monitoring its movement across successive frames in a video. It focuses on the temporal component of vision, answering not just the ‘what’ and ‘where’ of an object, but also tracking its trajectory over time. This is especially crucial in scenarios like traffic monitoring systems, where understanding the direction and speed of each vehicle is as important as identifying them. Tracking maintains the identity of an object across different frames, even when the object may temporarily disappear from view or get obscured.

Comparing the two, object detection is typically a one-off process in each frame and doesn’t consider the object’s history or future, whereas object tracking is a continuous process that builds on the initial detection. While detection is about recognizing and locating, tracking is about the continuity and movement of those recognized objects. In practical applications, these two technologies often work hand-in-hand: detection algorithms first identify objects in a frame, and tracking algorithms then follow these objects across subsequent frames. The synergy of both detection and tracking leads to robust and dynamic computer vision systems capable of understanding and interpreting real-world visual data in real time.

Types of Object Tracking

Image Object Tracking:

• Image object tracking, often referred to as single-frame tracking, involves identifying and tracking objects within a single still image.
• This type of tracking is particularly useful in applications where the object’s position and orientation need to be determined in a static context. For example, in augmented reality (AR) applications, image object tracking can be employed to superimpose digital information or graphics onto real-world objects in a single image.
• This is crucial for AR experiences where accurate alignment and placement of virtual elements on physical objects in the image are necessary, enhancing the user’s interaction with their environment.

Video Object Tracking:

• Video object tracking, on the other hand, extends the concept of tracking across multiple frames in a video sequence. This dynamic form of tracking is concerned with detecting and following objects as they move and change over time.
• It’s a more complex process due to factors like motion blur, changing lighting conditions, and occlusions. Video object tracking finds its use in numerous real-time applications, such as surveillance systems, where it’s crucial to monitor the movement of people or vehicles over time.
• For example, in a retail environment, video object tracking can be used to monitor customer flow and behavior, helping businesses optimize store layouts, and product placements, and even assess the effectiveness of marketing displays. This insight can be invaluable for enhancing the shopping experience and increasing sales efficiency.

Integration with Other Technologies:

• IoT Integration: In the Internet of Things (IoT), object tracking has become instrumental, particularly in smart home security systems. By fusing object tracking with IoT devices like cameras and sensors, these systems offer enhanced monitoring and security. For example, in a smart home, object tracking enables cameras to detect and follow unusual movements or identify known residents versus unknown individuals. This integration not only improves real-time surveillance but also aids in incident analysis, providing homeowners with both safety and peace of mind.
• AI and Machine Learning : Meanwhile, in the realms of Artificial Intelligence and Machine Learning, object tracking significantly bolsters predictive modeling, especially in retail analytics. Retail stores equipped with AI-driven cameras can utilize object tracking to analyze customer behaviors – from tracking foot traffic patterns to understanding how shoppers interact with products. This data feeds into ML algorithms, enabling retailers to optimize store layouts, and product placements, and even manage inventory more effectively based on customer behavior trends.
• Big Data: In the domain of Big Data, object tracking plays a critical role in processing and analyzing large-scale video data. In urban planning and traffic management, for instance, object tracking algorithms analyze hours of traffic footage to derive insights into traffic flow, congestion patterns, and accident hotspots. This integration allows for the processing of vast amounts of video data, transforming it into actionable insights that can inform policy decisions and urban infrastructure improvements. The confluence of object tracking with big data analytics leads to more informed decision-making and efficient management of resources in both the public and private sectors.

Implementing Object Tracking with YOLOv8 and DeepSORT

Step-by-Step Implementation

Below is a step-by-step guide to implement object tracking using YOLOv8 and DeepSORT .

Step 1: install the necessary libraries

These libraries are super important if you want to build a real-time object detection and tracking system. The deep-sort-realtime library is awesome because it uses the DeepSORT algorithm, which helps you accurately track multiple objects across video frames. It even uses deep learning to describe how the objects look, so it’s super accurate. The ultralytics library is also really cool because it has tools for working with YOLO models, specifically the YOLOv8. This means you can detect objects in images and videos efficiently. And lastly, the datetime module from Python’s standard library is really handy for managing dates and times. You can use it to calculate how long it takes to process each frame and figure out the frames per second (FPS) for performance measurement. When you put all these libraries together, you can do some seriously impressive object detection and tracking, which is crucial for analyzing dynamic videos.

pip install deep-sort-realtimepip install ultralyticspip install datetime

Input Video:

Step 2: Import Libraries

First, the necessary libraries are imported. The datetime library is used for handling date and time operations, which helps in measuring the processing time of each frame. The ultralytics library is used for loading and using the YOLO model, which performs object detection. The cv2 library (OpenCV) is employed for image and video processing, enabling reading, writing, and displaying video frames. The deep_sort_realtime library provides the DeepSORT tracker for object tracking, which keeps track of detected objects across frames. Lastly, the cv2_imshow function from the google.colab.patches module is imported for displaying images in Google Colab.

import datetime # Library for handling date and time operationsfrom ultralytics import YOLO # Library for loading and using the YOLO modelimport cv2 # OpenCV library for image and video processingfrom deep_sort_realtime.deepsort_tracker import DeepSort # Library for the DeepSORT trackerfrom google.colab.patches import cv2_imshow # Library for displaying images in Google Colab

Step 3: Define Helper Function

A helper function, create_video_writer, is defined to create a video writer object. This function takes the video capture object and output filename as inputs, retrieves the frame width, height, and FPS (frames per second) from the video capture object, and sets up the video writer with the MP4V codec. This writer object is used later to save the processed video frames.

def create_video_writer(video_cap, output_filename): # Function to create a video writer object for saving the output video frame_width = int(video_cap.get(cv2.CAP_PROP_FRAME_WIDTH)) frame_height = int(video_cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) fps = int(video_cap.get(cv2.CAP_PROP_FPS)) fourcc = cv2.VideoWriter_fourcc(*'MP4V') writer = cv2.VideoWriter(output_filename, fourcc, fps, (frame_width, frame_height)) return writer

Step 4: Initialize Parameters and Objects

Several parameters and objects are initialized. The CONFIDENCE_THRESHOLD is set to 0.8 to filter out low-confidence detections. Colors for drawing bounding boxes and text (green and white, respectively) are defined. The video capture object is initialized to read the input video file, and the video writer object is created using the helper function to save the output video. The YOLOv8n model is loaded for object detection, and the DeepSORT tracker is initialized to maintain object tracking across frames.

CONFIDENCE_THRESHOLD = 0.8 # Confidence threshold for detecting objectsGREEN = (0, 255, 0) # Color for drawing bounding boxesWHITE = (255, 255, 255) # Color for drawing textvideo_cap = cv2.VideoCapture("/content/video.mp4") # Initialize the video capture object to read the videowriter = create_video_writer(video_cap, "output.mp4") # Initialize the video writer object to save the processed videomodel = YOLO("yolov8n.pt") # Load the pre-trained YOLOv8n modeltracker = DeepSort(max_age=50) # Initialize the DeepSORT tracker

Step 5: Process Video Frames

Step 5.1: Start Frame Processing Loop

An infinite loop is started to process each frame of the video. The start time is recorded for performance measurement. A frame is read from the video capture object, and if no frame is read (indicating the end of the video), the loop is exited.

while True: start = datetime.datetime.now() # Record the start time ret, frame = video_cap.read() # Read a frame from the video if not ret: break # Exit the loop if no frame is read

Step 5.2: Run YOLO Model for Object Detection

The YOLO model is run on the current frame to detect objects. An empty list is initialized to store detection results. For each detected object, the confidence level is extracted and compared to the confidence threshold. Detections with confidence below the threshold are ignored. For valid detections, the bounding box coordinates and class ID are retrieved and added to the results list.

 detections = model(frame)[0] # Run the YOLO model on the frame to detect objects results = [] for data in detections.boxes.data.tolist(): confidence = data[4] # Extract the confidence level of the detection if float(confidence) < CONFIDENCE_THRESHOLD: continue # Ignore detections with low confidence # Get the bounding box coordinates and class ID xmin, ymin, xmax, ymax = int(data[0]), int(data[1]), int(data[2]), int(data[3]) class_id = int(data[5]) results.append([[xmin, ymin, xmax - xmin, ymax - ymin], confidence, class_id])

Step 5.3: Update Tracker with Detections

The DeepSORT tracker is updated with the new detections. The tracker maintains tracks for detected objects across frames. For each track, if it is confirmed, the track ID and bounding box coordinates are retrieved. Bounding boxes and track IDs are then drawn on the frame for visualization.

 tracks = tracker.update_tracks(results, frame=frame) for track in tracks: if not track.is_confirmed(): continue # Ignore unconfirmed tracks track_id = track.track_id # Get the track ID ltrb = track.to_ltrb() # Get the bounding box coordinates xmin, ymin, xmax, ymax = int(ltrb[0]), int(ltrb[1]), int(ltrb[2]), int(ltrb[3]) # Draw the bounding box and the track ID on the frame cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), GREEN, 2) cv2.rectangle(frame, (xmin, ymin - 20), (xmin + 20, ymin), GREEN, -1) cv2.putText(frame, str(track_id), (xmin + 5, ymin - 8), cv2.FONT_HERSHEY_SIMPLEX, 0.5, WHITE, 2)

Step 5.4: Calculate FPS and Display Frame

The end time is recorded, and the time taken to process the frame is calculated and printed. The frames per second (FPS) are calculated and displayed on the frame. The frame is displayed using cv2_imshow and written to the output video using the video writer object. If the ‘q’ key is pressed, the loop is exited, stopping the processing of frames.

 end = datetime.datetime.now() # Record the end time print(f"Time to process 1 frame: {(end - start).total_seconds() * 1000:.0f} milliseconds") fps = f"FPS: {1 / (end - start).total_seconds():.2f}" # Calculate and display the FPS cv2.putText(frame, fps, (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 2, (0, 0, 255), 8) # Display the frame and write it to the output video cv2_imshow(frame) writer.write(frame) if cv2.waitKey(1) == ord("q"): break # Exit the loop if 'q' key is pressed

Step 6: Release Resources

After processing all frames, the video capture and writer objects are released to free up resources, and all OpenCV windows are closed to clean up. This ensures that resources are properly managed and released after completing the video processing task

video_cap.release() # Release the video capture objectwriter.release() # Release the video writer objectcv2.destroyAllWindows() # Close all OpenCV windows

.

Output: