TECHNOLOGY

We move real space into 3D
and read the people in it with AI

We develop 3D Location Monitoring (digital twin) and visual AI ourselves and apply them to video security and automatic control

3D Location Monitoring Visual AI Detection features Movement path tracking Privacy-protecting display In-house 3D engine XRAS automatic control

3D Location Monitoring (digital twin)

We build a 3D virtual space that replicates the real one, and move the people and vehicles captured by the cameras into that space in real time

Building the space

The 3D space is built from floor plans, with drone footage used alongside them where needed
Surveillance areas are defined in three dimensions, so the spatial relationships between objects are taken into account and danger zones are judged precisely

Location synchronisation

Moving objects such as people in the video are synchronised into the 3D space in real time
The exact location of an incident is understood at once, which supports a proactive response

In normal operation XRTS shows only a background image with no people in it in the 3D space, and only the site where an abnormal event occurred is shown as live video, with the camera viewpoint and the viewer's viewpoint aligned

XRTS-based digital twin video security solution (AI Korea 2022, BEXCO)

XRTS-based video security solution for privacy protection (AI Expo 2024, COEX)

Visual AI

We detect moving objects such as people and vehicles in images and video, and estimate human posture

Object detectionA deep learning model specialised for video surveillance is refined by transfer learning for each site to find people and vehicles

Pose estimationThe positions of a person's joints are estimated and read together with their change over time to judge behaviours such as collapses, falls from height and caught-in events

Thermal imagingWe developed a model that detects people in thermal camera video as well (pilot on a maritime training ship)

Detection features

Abnormal events are detected and alerted in real time. Which features apply depends on the product and the site configuration

Virtual FenceCounts the objects that cross a line

Intrusion DetectionDetects an object entering a surveillance area

Loitering DetectionDetects an object staying and moving within a restricted area

Fall-from-Height DetectionDetects a person falling from height in a surveillance area

Collapse DetectionDetects a person who has collapsed

Caught-in DetectionDetects a body being caught in machinery

Privacy ProtectionDisplays people as skeletons or avatars to hide personally identifiable information (optional)

Facial RecognitionControls entry using facial information

Person TrackingTracks the location of a specified person in real time

Location TrackingConverts geospatial information into 3D to show object locations

Expected benefits

An alert is sent the moment an abnormal event occurs, so a response can begin immediately
Receiving the alert at the moment of detection allows the early signs of an incident to be confirmed
Real-time monitoring makes it possible to understand what is happening inside the building even when no one is at the desk
A large building can be managed effectively with limited staff
Safety accidents caused by high-risk equipment can be prevented and handled

Fields of application

Buildings and facilities that need restricted-access and danger-zone management
Construction sites and manufacturing plants where collapses and falls from height occur
Small business premises that need entry and loitering detection plus collapse detection during unstaffed hours
Manufacturing and construction, where high-risk equipment causes safety accidents

Movement path tracking

We track the movement paths of people and vehicles across multiple cameras in real time and show them on a flat map or a GIS-based 3D view (registered patents 10-2152318 · 10-2152319)

Expected benefits

Tracks the location of missing children and older people with dementia to prevent accidents in advance
Tracks the location of criminal suspects to block secondary crimes
Tracks the location of stolen or impounded vehicles to help resolve cases

Fields of application

Places with heavy foot traffic such as amusement parks and department stores
Government bodies that need to trace a suspect's movements
Local authorities responsible for public safety

Four privacy-protecting display methods

People in the video are displayed in different ways to match the security requirements of the site. De-identified display is an optional feature

Face MaskingBlurs faces in real time. If the AI misses a face it can be exposed, so the level of protection is the lowest

SkeletonReplaces people in the video with a skeleton rendering. Individuals cannot be recognised, and head, body, arms and legs are distinguished by colour

Pose EstimationRemoves the background as well and renders the person as a skeleton only. Suited to security-sensitive spaces

3D Location MonitoringRenders people as skeletons and builds the site as a 3D virtual space. Building the 3D space involves a cost

Personal Information Protection Commission preliminary adequacy review resolution (No. 2024-002-025)It is a resolution on our de-identified display method. It assumes that original video is displayed and recorded only for incident scenes, that originals of everyday scenes are not stored separately, that people on the monitoring screen are shown as icons or similar, and that the operating policy is disclosed to the data subjects in the filmed area

XRTS viewer screen — XRTS viewer — camera video and 3D space display

In-house 3D engine and system integration

In-house 3D rendering engineUsing an engine we built ourselves rather than a commercial game engine lets us adapt the process of projecting camera video into 3D space to each site. The rendering pipeline is optimised so it runs on low-specification PCs as well

XRTS configuration

Operator PC — synchronises the objects detected by the AI into the 3D space in real time and, when an event occurs, provides location awareness and a three-dimensional monitoring view
XRTS server — recognises objects in CCTV video, detects abnormal events such as loitering, intrusion, collapses, falls from height and caught-in events, and sends them over the network

Integration with existing systems

Supplied as a standalone application or installed as a plug-in to an existing video surveillance system
Technical support and joint development are available for turning another vendor's video surveillance view into 3D Location Monitoring

XRAS automatic control

Building and facility structures and devices such as HVAC units are reproduced in 3D space for monitoring and remote control

Monitoring and control

Devices inside a building or facility are visualised in 3D so their locations are grasped at a glance
Devices are monitored in real time and their settings changed by remote control
Control a device in the 3D space and the real device on site moves with it
Control equipment is integrated over standard protocols (Modbus · BACnet)

Data and energy

Device data is collected and shown as trends
Device information can be downloaded as a CSV file
AI analyses energy efficiency and forecasts energy consumption

XRAS HVAC 3D view — 3D monitoring of HVAC units

XRAS device control view — Remote device control

XRAS device location view — Device location awareness

We move real space into 3Dand read the people in it with AI