Laboratory for Analytical Sciences

I crafted and facilitated comprehensive testing sessions, formulated research methodologies, and executed interviews, task analysis, and usability testing. My contributions were then pivotal in driving enhancements for Elemendar’s AI-powered tool, READ.

CTI analysts manually process large volumes of intelligence reports, a method that is time-consuming and prone to cognitive overload. Elemendar developed READ, an AI-assisted tool designed to streamline entity annotation and threat network visualisation, but its effectiveness in real-world analysis workflows needed validation.

I facilitated two analyst workshops, including one for the UK Ministry of Defence, and one for LAS, where analysts completed intelligence tasks under AI-assisted and manual conditions. Task performance, questionnaires, and screen recordings were used to evaluate efficiency, cognitive load, and situational awareness. The findings showed faster report processing with AI but also highlighted frustrations with ML accuracy, and lack in confidence in ML annotations.

Analysis insights lead to improvements in custom rule creation, entity linking, and more transparency regarding ML annotation. Performance dashboard were created to give users transparency regarding ML annotations, and to visualisie improvements over time. Network Graph visualisations significantly improved analysts’ ability to connect cyber threats and draw intelligence conclusions.

Following a comprehensive evaluation of the enhancements implemented in both the user experience of the product and the data utilised for model training, we observed substantial advancements in performance when the same task was reassessed one year later with analysts from the Ministry of Defence.

AI tools enhance analyst efficiency but must balance automation with control to reduce frustration. These findings directly shaped READ.’s product development, ensuring it met real-world intelligence needs.