Video Solutions
The Limits of AI in Video Forensic Search
AI is reshaping video forensic search, turning hours of footage into minutes of insight. But expectations often outpace reality.
The “next wave” of video forensic search is underway, and AI is at the center of the evolution.
The volume of video that security teams are expected to review has grown far faster than the capacity to review it.
“A facility with dozens of cameras and a multi-hour incident window can represent hundreds of camera hours of footage, making manual timeline scrubbing not just inefficient but practically impossible for time-sensitive investigations,” says Chris Garner, senior product manager, Salient Systems, Austin, Texas.
The response from the VMS industry has been a shift toward attribute-driven search. “Rather than asking an operator to find the right camera and the right timestamp, modern forensic search tools let operators describe what they’re looking for — a person, a vehicle, a specific appearance — and surface relevant footage from across multiple cameras simultaneously,” Garner adds. “The underlying framework is intuitive: when did it happen, where should we look and what are we looking for? That shift from passive archive to active search tool is the most meaningful development in forensic search capability over the last several years.”
Many of these platforms and tools are now integrated with AI to make it possible to handle the sheer volume of video collected every day. Recently, SDM explored the technological advancements that have led to more comprehensive and effective video forensic search. AI tools are contributing to quicker, more accurate results for dealers and end users alike. As we continue to report on AI advancements, we’re exploring the “limitations” of AI — what it might realistically be able to do in the future, but can’t quite achieve yet.
“The underlying framework is intuitive: when did it happen, where should we look and what are we looking for? That shift from passive archive to active search tool is the most meaningful development in forensic search capability over the last several years.”
How Much AI Is Wishful Thinking?
End users’ demands for AI capabilities often err on the side of wishful thinking. “Clients often ask for capabilities like identifying ‘suspicious’ or ‘unusual’ behavior, which are inherently subjective and difficult to define consistently across different contexts,” says Tim Palmquist, vice president, Americas, Milestone Systems, Oswego, Ore. “Despite clear progress, there remains a gap between human intuition and what systems can reliably infer, especially in nuanced or ambiguous situations.”
Other end user demands are primarily centered around reliable, proactive, predictive and highly contextual understanding of complex events. But AI is only as good as its trainer and the data it’s provided with. “When you’re doing forensic search, you’re actually not searching the video; you’re searching the data that has been produced,” says Hans Kahler, chief operating officer, Brivo, Bethesda, Md. “There could be some limit of the analytics; it’s possible the analytics didn’t detect something or detected something incorrectly.”
Many AI models used are commercial off-the-shelf models that easily recognize common objects and routine events — which sounds great as a baseline, but security operators are often focused on rare or nuanced situations, where these generic models do not perform as well. “Models individually tuned for security and safety use cases have dramatically better performance,” says Matt Cirnigliaro, head of product marketing – North America, IQSIGHT, Fairport, N.Y. “Therefore, when selecting a VMS platform, research and testing are important to evaluate how well the solution performs against the organization’s most common and critical use cases.”
Looking for quick answers on security topics? Try Ask SDM, our new smart AI search tool. Ask SDM →
This takes a lot of work — because, again, AI is only as good as what it’s supplied with, and every end user is looking for unique capabilities tailored to their needs. The “CSI effect” has taken some consumers by storm, as end users are frequently misinformed about what forensic search capabilities can actually do. Some think it’s as easy to reconstruct events as it is on TV. But not all cameras have such great quality, nor will they always accurately capture license plates or faces to where they’re perfectly identifiable. AI, unfortunately, cannot help with this.
“Forensic search is entirely downstream of what the camera detects and reports; if the camera’s AI is poorly trained, misconfigured or doesn’t report a given attribute, the VMS has nothing to work with,” Garner says. “This affects every search on every camera, and customers often don’t recognize it as the source of poor results, attributing missed detections to the search tool rather than the upstream metadata.”
Additionally, AI models only understand a moment in time versus chronological context. End users commonly request seamless cross-camera subject tracking. “Customers want to describe an incident and have the system surface correlated results across video, access control, sensor triggers and other data sources in a single query,” Garner says. “That experience is only as reliable as the metadata each source provides, and inconsistency across systems is the barrier that has to be solved before it becomes practical.”
“When you’re doing forensic search, you’re actually not searching the video; you’re searching the data that has been produced. There could be some limit of the analytics; it’s possible the analytics didn’t detect something or detected something incorrectly.”
AI Event Reconstruction Isn’t a Possibility Today
Like most (if not all) other areas of video surveillance, the “next wave” of video forensic search is a shift from reactive object detection to proactive, contextual understanding and predictive analytics. “This shift will be possible due to a combination of generative AI, advanced cloud architectures and deepfake detection,” says Wayne Davis, solutions engineer, Axis Communications, Chelmsford, Mass. “When this shift happens, generative AI could reconstruct events and simulate scenarios, cloud architectures will enable scalable processing and advanced models and deepfake detection will ensure video integrity.”
The topic of event reconstruction is a hot one — because it’s a great capability in theory, but not one that AI is ready to take over yet. At most, current systems can significantly accelerate investigations by narrowing down relevant footage and reducing manual review. Actually reconstructing complex events, especially across multiple cameras, still requires human oversight because events are not just single moments; they unfold in a specific order over a period of time. “Achieving this capability will depend on advances in cross-camera reasoning, temporal understanding and robust validation mechanisms,” Palmquist says. “Additionally, incorporating domain expertise and enabling systems to learn over time from operator feedback will be critical to improving accuracy and reliability in real-world deployments.”
For scoped investigations with good camera coverage and a known time window, attribute-based search has compressed investigation time from hours to minutes. Full automated reconstruction is harder. “The core unsolved problem isn’t motion or detection; it’s object re-identification,” Garner says. “The breakthroughs needed are more robust re-identification under environmental variability and consistent metadata quality across a wider range of camera hardware.”
Clearly, significant breakthroughs are required to get to a place where AI could reliably reconstruct an event.
It isn’t just the limitations of AI that prevent accurate event reconstruction; it’s also, as mentioned previously, the quality and availability of data. “These [breakthroughs] include advanced causal reasoning and common-sense AI to understand ‘why’ events occur; robust multi-modal data fusion for seamless semantic interpretation across disparate sources; reliable anomaly detection for unforeseen events; and contextual memory for continuous learning,” Davis says. “Ethical AI and bias mitigation are also crucial to ensure transparent and trustworthy reconstructions.”
Combatting Biases in AI Forensic Search
As AI capabilities evolve, it’s imperative to watch for bias, such as misidentification and missed events, in AI-driven search results. “It is crucial to recognize that AI, while a powerful tool, is not a complete solution on its own,” says Wayne Davis of Axis Communications. “These ‘hiccups’ often arise from training data limitations, where insufficient diversity or skewed representations can lead to misclassification.”
Consider, for example, an AI model trained to detect a soccer ball but only exposed to images of balls on green grass or turf. “The model may associate the green background as important to identifying an object as a soccer ball,” says Matt Cirnigliaro of IQSIGHT. “As a result, it may struggle to identify the same object on blue turf or other surfaces. This type of bias must be addressed through comprehensive training data.”
Many AI models are optimized for specific conditions, and they may perform poorly outside of these parameters. “To address these challenges, we adopt a multi-faceted approach, with the most critical strategy being the integration of a human-in-the-loop, where AI assists in narrowing down data, but final verification and decision-making rest with the human operator,” Davis says.
Axis also emphasizes system-wide optimization, recommending high-quality cameras and strategic placement to provide AI with optimal data. “We understand that solid detection often requires specific preconditions like sufficient time in the scene or good contrast,” Davis says. “Additionally, we engage in continuous improvement through feedback loops, monitoring model performance in real-world deployments to refine and retrain AI models, aiming to reduce bias and improve accuracy, while also striving for greater transparency and explainability in AI’s conclusions.”
Milestone Systems similarly focuses on continuous validation against real-world data to ensure models perform reliably in operational conditions. “We also combine multiple signals and models rather than relying on a single source of truth, which helps improve robustness and reduce systematic errors,” says Tim Palmquist of Milestone. “Just as importantly, we keep the user actively involved in the loop, enabling oversight and correction when needed. The objective is not to completely eliminate bias — which is unrealistic — but to create systems that are transparent, well-understood and give users control over how results are interpreted and used.”
AI-Driven Behavioral Analysis Is a Work in Progress
There are mixed feelings about whether or not AI can understand intent. Humans have a “gut feeling,” sometimes; anyone who has worked in retail, for example, can often just tell when someone is walking into the store with the intent to steal something. AI doesn’t have that capability yet — or does it?
“AI is clearly moving in the direction of understanding intent and behavior, but this transition will be gradual,” Palmquist says. “Interpreting intent requires combining multiple signals across time and context, making it significantly more complex than object detection. AI will increasingly assist by suggesting interpretations and highlighting patterns, but human validation will remain essential to ensure accuracy and accountability, especially in critical scenarios.”
There are certain behaviors that AI can currently detect, i.e. loitering, running, abandoned objects, tailgating and more. These capabilities are already available in camera-side analytics and surfaced through VMS search. “True intent inference is harder and warrants careful framing,” Garner says. “Flagging anomalous behavior patterns is reasonable, but asserting intent from video data alone carries significant risk of false positives and raises ethical concerns. The realistic near-term evolution is richer behavioral filters as search criteria — finding ‘a person who stopped in this area for more than five minutes’ rather than inferring why they stopped. That’s operationally valuable without overreaching.”
Overall, the evolution from AI detecting objects to actually understanding intent and behavior is still a work in progress — but security leaders generally have confidence in the fruition of this evolution. “This progression is driven by AI’s increasing ability to recognize complex patterns in space and time, inferring higher-level behaviors from sequences of actions and interactions,” Davis says. “Multi-modal sensor integration (e.g., radar, audio) provides richer context, allowing AI to move beyond isolated video analysis. Furthermore, generative AI influences predictive capabilities, shifting from identifying ‘a person’ to understanding ‘a person engaging in suspicious activity,’ leading to more proactive security outcomes.”
What AI can do today is on the simple side of the equation and still requires some human oversight. GenAI solutions are actively leveraging cloud-based large language models to add context and reasoning to what video security cameras capture, transcending basic object detection by interpreting complex environments to provide insights to the user. With simple prompts, GenAI-enabled video solutions can generate detailed descriptions of events or scenes, like a human operator, without requiring training on specific objects or behaviors. This capability enables proactive responses, saving time and resources. “For example, when loitering is detected, video solutions that use GenAI can supplement this detection with an additional layer that provides further insight into the situation,” Cirnigliaro says. “It could inform that a person stopped briefly to tie their shoe, is actively spraying graffiti on an external wall, is attempting to break into a vehicle in a parking lot or appears hurt or in distress. The operator can then respond appropriately to the specific situation.”
“AI is clearly moving in the direction of understanding intent and behavior, but this transition will be gradual.”
Tim Palmquist, Milestone Systems
Leaders are confident that the “next wave” of video forensic search will be built by AI capabilities. Kahler says cloud architecture and GenAI combined with natural language will continue to drive growth in this segment. “You’ll basically have an integration into ChatGPT or your favorite AI bot and type in: ‘What happened in my building yesterday?’” he says. “You could even set it up to look on a daily basis: ‘Give me a summary every morning of what happened yesterday.’”
Palmquist agrees, “Systems will evolve from detecting objects to interpreting events and behaviors, allowing users to interact with video more like questioning a witness — asking what happened, why and how events unfolded. While generative AI and VLM/LLM technologies will play a key role, their value will be maximized when grounded in real-world video data and constrained by physical context. This shift will transform video systems into more conversational, intelligent tools capable of supporting higher-level investigative reasoning.”
These capabilities are designed, ultimately, to make the most of existing security systems and enhance operational efficiency. As AI continues to evolve, personnel in every corner of the security ecosystem will reap the benefits of more seamless and proactive security solutions.
Looking for a reprint of this article?
From high-res PDFs to custom plaques, order your copy today!
