What is OCR and How Does it Works?
Teramind’s Optical Character Recognition or OCR feature allows you to discover onscreen Personally Identifiable Information (PII), Protected Health Information (PHI), Private Financial Information (PFI), and other sensitive information using keywords, patterns, and regular expressions.
Teramind’s proprietary OCR engine continuously captures, indexes, and analyses a user’s desktop using machine learning and sophisticated pattern recognition algorithms.
With OCR, you can quickly search for textual information displayed on the screen; even inside images or videos for forensic investigation purposes. You can also build powerful activity rules and automatically get notified when a piece of information is displayed on the user’s screen.
The OCR works with multi-monitor setups and virtual desktops, including the ones from the Terminal Servers/RDP sessions.
Some use cases of Teramind’s OCR features are:
Alert when a user sees a full credit card number on the screen violating the PCI DSS compliance requirements.
Find out how often a user saw certain sensitive records and recall desktop videos & screenshots of those occurrences for investigation purposes.
Create rules for applications that are not easily parseable because the text is inside images or videos.
Notify an admin when certain text appears on the screen.
Prevent embargoed data exposure such as M&A announcement, stop insider trading, etc.
Monitor the activity of third-party vendors, contractors, and other external users on a Terminal Server without needing to install an Agent.
OCR Search
Teramind automatically analyzes any content displayed on the user’s screen in the background and creates an index of text it detects. This allows you to conduct a high-speed search of any onscreen text using the OCR search feature.
You can access the OCR search screen from the MONITORING > OCR menu:
You can use Full Text, Regular Expressions*, Wildcards, or Contains (phrases) for your search terms.
*Please note that Teramind supports the Elasticsearch regular expression syntax for OCR search and OCR rules. More information can be found about it in Elastic documentation.
From the search results, you can click the Camera icon to view a screen snapshot. The snapshot will show the user’s desktop with the areas highlighted in semi-transparent Yellow color where the OCR text was detected:
You can also use the Movie Camera icon to view a session recording of the exact date/time when the text appeared on the screen.
Useful Resources:
OCR Rules
You can use OCR-based Activity rules to detect text on screen(s). You can optionally limit the detection to some specific applications, specify detection thresholds (e.g., how many times the text was detected in a day) and risk levels:
Please note that Teramind supports the Elasticsearch regular expression syntax for OCR search and OCR rules. More information can be found about it in Elastic documentation.
Useful Resources:
What is Session Mining? What are OCR Mining Node and OCR Database Node?
You will hear these terms when working with OCR in Teramind. Basically, session mining means capturing the screens from the user desktop and storing them in the server for later processed by the OCR engine.
The OCR engine uses two nodes/servers. The Mining Node runs the OCR engine which analyzes and identifies texts from the screen recordings. The Database Node (Elasticsearch) facilities the storing and searching of the text. You will need to set up both of these nodes for the OCR to work properly in your on-premise or private cloud deployments (see below). For Cloud deployments, the nodes will be set up and managed by us.
You can see the nodes displayed under the Nodes section on the Settings > Server Management screen:
Useful Resources:
OCR Server Requirements
If you are using the on-premise or private cloud deployment, you need to set up at least one OCR Database Node and one Mining Node for the OCR features to work. Additional nodes may be required for larger deployments of 200 or more users.
Generally speaking, the more powerful a server is, the better performance you will get. See the OCR Performance and Mining Delays section below for more information.
The deployment guides below have the detailed specifications for OCR node sizes, CPU and RAM requirements for each deployment scenario.
Useful Resources:
How to Set Up OCR for Your Deployment
Cloud Deployment
If it is a Cloud trial, send an email to [email protected].
On-Premise Deployment
You will need to set up the OCR nodes in your deployment. Follow the instructions here to set up OCR for your on-premise deployment.
AWS Deployment
You will need to set up the OCR nodes in your deployment. Follow the instructions here to set up OCR for your AWS deployment.
Azure Deployment
You will need to set up the OCR nodes in your deployment. Follow the instructions here to set up OCR for your Azure deployment.
Enabling OCR on the Monitoring Settings
You can enable/disable OCR from the Monitoring Settings > Monitoring Profile screen. You can change the OCR language and processing start date from the Monitoring Settings > Monitoring Profile > OCR window:
Screen Monitoring Settings and OCR
Some of the Screen monitoring settings may affect the OCR behavior or make it unstable.
For example, update screen on events only, no. of frames per second, etc. will affect the accuracy of the OCR detection.
If you use the RECORD ONLY WHEN BEHAVIOR RULE WAS VIOLATED option, then the OCR will not be able to capture texts for the entire duration of the user sessions.
If you have DELETE HISTORY AFTER (DAYS) settings to auto-delete recordings after certain days; the OCR search will be able to find previous texts it captured but will not be able to show the screen snapshot or the video recording of the time.
If use the ASYNC SCREEN UPLOAD option, it will force Teramind to use a queue for screen recordings instead of uploading them in real-time. It is suitable for a slower network or a busy OCR server. However, you might experience some delay between the user activity and the recording appearing on the dashboard when ASYNC is enabled.
Also, do not use OCR feature simultaneously with LIVE SCREEN SCALING (less than 100%), otherwise there might be performance and accuracy issues.
Useful Resources:
OCR Performance and Mining Delays
The OCR, due to how it works, is not real-time. So, sometimes the OCR search will not find information if the OCR engine has not processed the screen recording yet. Or, an OCR-based rule might trigger slightly later than when an action actually took place. This is why you will notice that the OCR only supports the Notify action.
Several factors might affect the performance of the OCR processing. The number of screens, no. of users, user’s activity level, the OCR node configurations, network bandwidth, etc. Usually adding more CPU/RAM and disk space can increase the performance of the OCR engine.
You can view how your OCR servers are performing from the Settings > System Health screen. Under the Session mining stats you will see the OCR processing status:
