KR20240036761A - Safety management system based on deep learning and method thereof - Google Patents

Abstract

The present invention relates to a deep learning-based safety management system, which comprises: a camera module photographing an image of an area to be monitored in an industrial site; and a processor receiving and analyzing image data photographed by the camera module in real time, and performing a response operation suitable for a situation of the analyzed industrial site.

Description

Deep learning-based safety management system and method {SAFETY MANAGEMENT SYSTEM BASED ON DEEP LEARNING AND METHOD THEREOF}

The present invention relates to a deep learning-based safety management system and method. More specifically, a deep learning-based safety management system and method that enables safety management in industrial sites through analysis of real-time video data based on deep learning. It's about.

Safety management systems in conventional industrial sites require the installation of various sensors to detect worker entry into a specific area or fire.

For example, the above sensors are representative of area sensors for detecting specific areas and temperature sensors for detecting sudden temperature changes due to fire. In the existing safety management system as described above, sensors for each necessary function are additionally installed. It is required.

However, if installation of the relevant sensors is not possible due to the specificity of the site, it is difficult to apply it to the safety management system.

For example, when trying to detect a worker's intrusion (entry) into an area where a robot is moving, application of an area sensor may be difficult due to the robot's movement. Additionally, installation and periodic management of the sensors costs money, and this cost increases as functions are added.

Therefore, in terms of flexibility for applying new functions to industrial sites (e.g., new monitoring functions can be easily added in software without adding hardware sensors) and cost for installation and maintenance, it is possible to install or install sensors. There is a need for technology that allows industrial safety management to be carried out without any need, just by analyzing video data acquired by cameras.

The background technology of the present invention is disclosed in Korean Patent Publication No. 10-2022-0077404 (published on June 9, 2022, multi-level work site safety control method and system for worker safety).

According to one aspect of the present invention, the present invention was created to solve the above problems, and is a deep learning-based safety system that enables safety management in industrial sites through analysis of real-time image data based on deep learning. The purpose is to provide management systems and methods.

A deep learning-based safety management system according to one aspect of the present invention includes a camera module that captures images of an area to be monitored at an industrial site; and a processor that receives and analyzes video data captured by the camera module in real time and performs corresponding actions appropriate for the analyzed industrial site situation.

In the present invention, the processor is characterized by previously performing learning to analyze the image data based on cascaded deep learning.

In the present invention, at least one or more of the image data, an object recognition model, a worker's posture estimation model, a worker's motion classification model, response alarm or response action data for each incident, and response alarm or response action data for each detected motion are stored. and a database, wherein the processor performs analysis of the image data with reference to information stored in the database.

In the present invention, the processor is characterized in that it outputs a response alarm or response action command to the target device based on the analysis result of the image data.

In the present invention, an alarm module that outputs a corresponding alarm for each accident or detection operation under the control of the processor is characterized in that it further includes.

In the present invention, the camera module inputs video, which is real-time video data, to the processor in units of frames using RTSP (Real Time Streaming Protocol).

In the present invention, the processor applies an object recognition model to the image data to detect the occurrence of a worker or a designated accident and output a response alarm, and the designated accident is at least one of fire, flash, and smoke. It is characterized by including.

In the present invention, when workers are recognized through an object recognition model, the processor applies a posture estimation model to the recognized workers to calculate the positions of major body parts or joints corresponding to the posture of each worker. It is characterized by

In the present invention, the processor, based on the result of performing posture estimation, detects the intrusion of a worker into the area where the equipment or robot is operating, in order to prevent the occurrence of a safety accident. It is characterized by stopping the operation of.

In the present invention, the processor estimates the workers' movements by applying a motion classification model to the recognized workers, and when dangerous movements of the workers are detected as a result of estimating the workers' motions, through an alarm module. It outputs a response alarm, classifies it into a plurality of main actions that can be commonly taken in each industrial site, and detects lying down and falling out of the plurality of main actions as dangerous actions.

A deep learning-based safety management method according to another aspect of the present invention includes capturing an image of an area to be monitored at an industrial site through a camera module and inputting it to a processor; And a step of the processor receiving and analyzing image data captured by the camera module in real time, and performing a corresponding action appropriate to the analyzed situation of the industrial site.

In the present invention, at least one of the image data, object recognition model, worker's posture estimation model, worker's motion classification model, response alarm or response action data for each incident, and response alarm or response action data for each detected motion through the database. It may further include a step of storing the abnormality, wherein the processor performs analysis of the image data with reference to information stored in the database.

In the present invention, in the step of performing a response operation appropriate to the analyzed situation of the industrial site, the processor outputs a response alarm or response action command to the target device based on the analysis result of the image data. do.

In the present invention, in the step of capturing the video and inputting it to the processor, the camera module inputs video, which is real-time video data, to the processor in units of frames using RTSP (Real Time Streaming Protocol).

In the present invention, in the step of performing a response operation appropriate to the analyzed situation of the industrial site, the processor detects the occurrence of a worker or a designated accident by applying an object recognition model to the image data and outputs a response alarm. , the specified accident is characterized in that it includes at least one of fire, flash, and smoke.

In the present invention, in the step of performing a response operation appropriate to the analyzed industrial site situation, the processor, when workers are recognized through an object recognition model, applies a posture estimation model to the recognized workers. , It is characterized by calculating the positions of major body parts or joints corresponding to the posture of each worker.

In the present invention, in the step of performing a response operation appropriate to the analyzed situation of the industrial site, the processor detects the intrusion of a worker into the area where the equipment or robot is operating, based on the result of performing posture estimation. If possible, the operation of the equipment or robot in operation is stopped in order to prevent safety accidents.

In the present invention, in the step of performing a response operation appropriate to the analyzed situation of the industrial site, the processor estimates the movements of the workers by applying a movement classification model to the recognized workers, and determines the movements of the workers. As a result of the estimation, if dangerous movements of workers are detected, a response alarm is output through the alarm module, but it is classified into a plurality of main actions that can be commonly taken at each industrial site, and among the plurality of main movements, lying down and falling are It is characterized by detecting as a dangerous operation.

According to one aspect of the present invention, the present invention enables safety management at industrial sites through analysis of real-time image data based on deep learning.

1 is an exemplary diagram showing the schematic configuration of a deep learning-based safety management system according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a deep learning-based safety management method according to an embodiment of the present invention.
FIG. 3 is an example diagram shown in FIG. 2 to explain the deep learning algorithm that performs situation judgment by combining the results recognized through the object recognition model, posture estimation model, and motion classification model.
Figure 4 is an example diagram shown in Figure 2 to explain the method of estimating the movements of workers and outputting a response alarm appropriate for the situation when dangerous movements are detected.

Hereinafter, an embodiment of a step-by-step deep learning-based safety management system and method according to the present invention will be described with reference to the attached drawings.

In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

Figure 1 is an exemplary diagram showing the schematic configuration of a deep learning-based safety management system according to an embodiment of the present invention.

As shown in FIG. 1, the deep learning-based safety management system according to this embodiment includes a camera module 110, a processor 120, a database 130, and/or an alarm module 140.

The camera module 110 captures images of the area to be monitored.

A plurality of camera modules 110 may be installed at an industrial site.

The processor 120 receives and analyzes image data captured by the camera module 110 in real time.

The processor 120 performs learning to analyze the image data based on cascaded deep learning.

The database 130 includes the image data, object recognition model, worker's posture estimation model, worker's motion classification model, response alarm or response action data for each accident (e.g., fire, flash, smoke, etc.), and response for each detected motion. Save at least one of alarm or response action data.

At this time, the database 130 may be replaced with the internal memory (not shown) of the processor 120.

The processor 120 may perform analysis of the image data with reference to information stored in the database 130, and may send a response alarm and/or response action command to the target device (e.g. : Equipment, robots, etc.) can be output.

The alarm module 140 outputs a corresponding alarm (eg, a visual alarm, an auditory alarm, a tactile alarm) for each accident occurrence or detection operation under the control of the processor 120. In addition, the signal output from the alarm module 140 (i.e., response alarm for each accident or detection operation) is transmitted to a designated device (eg, PLC, Programmable Logic Controller) to stop the designated facility or robot.

Hereinafter, the operation of the step-by-step deep learning-based safety management system according to this embodiment will be described with reference to the flowchart of FIG. 2.

Figure 2 is a flowchart illustrating a deep learning-based safety management method according to an embodiment of the present invention.

Referring to FIG. 2, the processor 120 receives image data (e.g., video) from the camera module 110 (e.g., IP camera) in real time using RTSP (Real Time Streaming Protocol) (S101).

For example, the processor 120 receives video in real time from the camera module 110 in units of frames using the RTSP method.

For reference, the most important factor in real-time safety management is to be able to quickly recognize an incident when it occurs and take immediate action. Accordingly, the deep learning algorithm applied in this embodiment satisfies the two criteria of high accuracy and short execution time (Inference Time). In addition, in order to ensure that video frames transmitted through the camera module 110 are continuously transmitted to the processor 120 without delay, the video collected by the camera module 110 is transmitted via the network using Real Time Streaming Protocol (RTSP). do. Additionally, due to the nature of the system connecting a plurality of cameras to each processor 120, there is an advantage in that the camera modules 110 can be easily integrated and managed.

In addition, the processor 120 applies an object recognition model to the image data (i.e., video frame) to detect the occurrence of an operator and/or a designated accident (e.g., fire, flash, smoke, etc.) (S102, S103, S105) ).

For example, referring to FIG. 3, the processor 120 recognizes four elements, namely Person, Fire, Flash, and Smoke, from the frame through an object recognition model (i.e., object recognition deep learning model). Classify the branch objects and specify the location (Localization) as a rectangle (Bounding Box). At this time, this embodiment selects and applies the YOLOv5 (You Only Look Once) model that satisfies the two criteria of high accuracy and short execution time (Inference Time). The YOLOv5 model can secure over 99% performance through optimization despite its very fast execution time. At this time, even if the video frame size is different, the video frame is resized to 608 By doing so, there is almost no performance degradation due to this.

If at least one of the specified accidents (e.g., fire, flash, smoke, etc.) is detected, a response alarm is output through the alarm module 140 (S104).

Additionally, when workers are recognized through the object recognition model, the processor 120 applies a pose estimation model (i.e., a pose estimation deep learning model) to the recognized workers to determine the pose of each worker. Calculate the positions of major body parts or joints (e.g., 17) corresponding to (S106).

For example, the posture estimation model performed on the recognized workers is a core function that enables detailed inspection of whether the worker is intruding (entering) into a prohibited area and prediction of the worker's motion. Multiple (e.g. 17) major body parts or joints (nose, left eye: L-Eye, right eye: R-Eye, left ear: L-Ear, right ear: R-Ear, left shoulder: L-Shoulder, Right Shoulder:R-Shoulder, Left Elbow:L-Elbow, Right Elbow:R-Elbow, Left Wrist:L-Wrist, Right Wrist:R-Wrist, Left Hip:L-Hip, Right Hip:R -Hip, left knee: L-Knee, right knee: R-knee, left wrist: L-Ankle, right wrist: R-Ankle) location is estimated. At this time, accurate coordinate estimation of each body part or joint can detect the worker's intrusion into the area in more detail and also enables recognition of specific parts. In this embodiment, the pose estimation model is HRNET (Deep High Resolution Network) and includes a model that has achieved SOTA (State of the Art) in the field of Human Pose Estimation. Referring to Figure 3, the areas recognized as each worker (Bounding Box) are cropped, resized to 256 x 192, and a posture estimation model is applied to ensure high accuracy and fast execution time.

In addition, based on the result of performing the posture estimation, the processor 120 detects the worker's intrusion (entry) into a prohibited area (e.g., an area where equipment or a robot is moving) ( Example of S107), to prevent safety accidents, the operation of equipment or robots operating within the relevant area is stopped (S108).

For example, the location coordinates of multiple (e.g. 17) major body parts or joints calculated for each worker not only enable more precise intrusion detection compared to existing methods, but can also be used to classify the worker's current movement.

For reference, this embodiment is a motion classification model that analyzes the motions of each worker using FCL (Fully-connected Layers) to identify multiple (e.g. 10) main motions (e.g., standing) that can be commonly taken in each industrial site. (standing up, sitting down, walking, running, jumping, lying down, bending/picking, carrying loads, reaching up) ), falling down). Among the above movements, lying down and falling down are classified as dangerous movements because they cause the worker to fall.

Additionally, the processor 120 applies a motion classification model to the recognized workers (e.g., applies a Fully Connected Layer (FCL) model to the workers' major body parts or joint positions) to estimate the workers' motions. (S109).

As a result of estimating the workers' movements, if the workers' dangerous behavior is detected (example in S110), a response alarm appropriate for the situation is output through the alarm module 140 (S111) (see FIG. 4) .

As described above, this embodiment synthesizes the results recognized through three deep learning algorithms (e.g., object recognition model, posture estimation model, and motion classification model) to respond appropriately to each situation. For example, it notifies workers and managers of recognized situations and allows them to take quick response actions.

At this time, the judgment criteria and response measures for each dangerous situation are configured according to the industrial site, especially the generation of equipment alarms (e.g., outputting an alarm in equipment where the worker's dangerous behavior can be applied to stop the worker's dangerous behavior). It includes three response actions: stopping the robot and/or equipment (e.g., stopping the robot and/or equipment when the operator's entry cannot be stopped), and generating a critical alarm. Here, facility alarms and robot stop actions correspond to local responses corresponding to the area where each situation occurs, and critical alarm actions correspond to global responses to the entire factory, such as fire.

As described above, this embodiment receives image data (e.g., video) collected through the camera module 110 in real time, and performs object detection on the received image data (e.g., video). Detects (recognizes) pre-designated objects (e.g. workers, fire, flashes, smoke, etc.), and uses Pose Estimation for the detected (recognized) workers to determine the positions of their main body parts or joints (e.g. : 17) are calculated, the worker's movements are detected (recognized) through the calculated major body parts or joint positions, and the detected (recognized) results are comprehensively judged to provide corresponding actions appropriate for the situation in the industrial site. (Example: alarm, robot stop, etc.).

As described above, this embodiment detects worker entry into a hazardous area or fire through video data analysis, and takes appropriate response measures according to the situation at the industrial site determined through analysis of video data without the need to install or install sensors. It has the effect of enabling it to be implemented.

Additionally, this embodiment can reduce the number of sensors that need to be installed and/or maintained to ensure safety in industrial sites, and, unlike sensors that have limitations in recognition due to simple functions, more accurate and detailed safety diagnosis is possible. This has the effect of allowing safety management to be performed using cameras even in areas where sensors cannot be installed due to the special characteristics of industrial sites.

As described above, the present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and various modifications and equivalent embodiments can be made by those skilled in the art. You will understand the point. Therefore, the scope of technical protection of the present invention should be determined by the scope of the patent claims below. Implementations described herein may also be implemented as, for example, a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device that includes a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

110: camera module
120: processor
130: database
140: Alarm module

Claims (18)

A camera module that captures images of the area to be monitored in an industrial site; and
A deep learning-based safety management system comprising a processor that receives and analyzes video data captured by the camera module in real time and performs response actions appropriate for the analyzed industrial site situation.
The method of claim 1, wherein the processor:
A deep learning-based safety management system characterized by performing learning in advance to analyze the image data based on cascaded deep learning.
According to clause 1,
A database that stores at least one of the image data, an object recognition model, a worker's posture estimation model, a worker's motion classification model, response alarm or response action data for each incident, and response alarm or response action data for each detected action; Contains,
The processor,
A deep learning-based safety management system, characterized in that analysis of the image data is performed with reference to information stored in the database.
The method of claim 1, wherein the processor:
A deep learning-based safety management system that outputs a response alarm or response action command to the target device based on the analysis results of the video data.
According to clause 1,
A deep learning-based safety management system further comprising an alarm module that outputs a response alarm for each accident or detection operation under the control of the processor.
The method of claim 1, wherein the camera module:
A deep learning-based safety management system characterized by inputting video, which is real-time video data, into the processor in frame units using RTSP (Real Time Streaming Protocol).
The method of claim 1, wherein the processor:
An object recognition model is applied to the image data to detect the occurrence of a worker or a designated accident and output a response alarm,
The accidents specified above are:
A deep learning-based safety management system comprising at least one of fire, flash, and smoke.
The method of claim 1, wherein the processor:
When workers are recognized through an object recognition model,
A deep learning-based safety management system that applies a posture estimation model to the recognized workers and calculates the positions of major body parts or joints corresponding to the posture of each worker.
The method of claim 1, wherein the processor:
Based on the results of posture estimation, if a worker's intrusion is detected into the area where the equipment or robot is operating,
A deep learning-based safety management system characterized by stopping the operation of the operating equipment or robot in order to prevent safety accidents.
The method of claim 1, wherein the processor:
A motion classification model is applied to the recognized workers to estimate the workers' motions, and when dangerous motions of the workers are detected as a result of estimating the workers' motions, a corresponding alarm is output through the alarm module,
A deep learning-based safety management system that classifies a plurality of main actions that can be commonly taken at each industrial site and detects lying down and falling among the plurality of main actions as dangerous actions.
A step of capturing images of an area to be monitored at an industrial site through a camera module and inputting the images to a processor; and
A deep learning-based safety management method comprising: receiving and analyzing video data captured by the camera module in real time by a processor, and performing response actions appropriate for the analyzed situation at an industrial site.
According to clause 11,
Storing at least one of the image data, object recognition model, worker's posture estimation model, worker's motion classification model, response alarm or response action data for each incident, and response alarm or response action data for each detected motion through a database. further includes ;,
The processor,
A deep learning-based safety management method, characterized in that analysis of the image data is performed with reference to information stored in the database.
The method of claim 11, wherein in the step of performing a response operation appropriate to the analyzed situation of the industrial site,
The processor,
A deep learning-based safety management method characterized by outputting a response alarm or response action command to the target device based on the analysis results of the video data.
The method of claim 11, wherein in the step of capturing the image and inputting it to the processor,
The camera module is,
A deep learning-based safety management method characterized by inputting video, which is real-time video data, into the processor in frame units using RTSP (Real Time Streaming Protocol).
According to claim 11,
In the step of performing response actions appropriate to the situation of the industrial site analyzed above,
The processor,
An object recognition model is applied to the image data to detect the occurrence of a worker or a designated accident and output a response alarm,
The accidents specified above are:
A deep learning-based safety management method comprising at least one of fire, flash, and smoke.
According to clause 11,
In the step of performing response actions appropriate to the situation of the industrial site analyzed above,
The processor,
When workers are recognized through an object recognition model,
A deep learning-based safety management method characterized by applying a posture estimation model to the recognized workers and calculating the positions of major body parts or joints corresponding to the posture of each worker.
According to claim 11,
In the step of performing response actions appropriate to the situation of the industrial site analyzed above,
The processor,
Based on the results of posture estimation, if a worker's intrusion is detected into the area where the equipment or robot is operating,
A deep learning-based safety management method characterized by stopping the operation of the operating equipment or robot in order to prevent safety accidents.
According to clause 11,
In the step of performing response actions appropriate to the situation of the industrial site analyzed above,
The processor,
A motion classification model is applied to the recognized workers to estimate the workers' motions, and when dangerous motions of the workers are detected as a result of estimating the workers' motions, a corresponding alarm is output through the alarm module,
A deep learning-based safety management method characterized by classifying a plurality of main actions that can be commonly taken at each industrial site, and detecting lying down and falling among the plurality of main actions as dangerous actions.

Images