GAICo-D: Exploring Commercialization of Domain-Aware Generative AI Evaluation

Project Overview

This I-Corps project builds on an evaluation framework for comparing results from Generative Artificial Intelligence (GenAI) systems. Organizations are rapidly trying out Foundation Models (FMs) and Large Language Models (LLMs) to support complex data analyses in domains like health, finance, law, and chemical manufacturing. These models promise improved performance and flexibility across diverse recording modalities. However, practitioners today often resort to ad hoc, non-standardized scripts using their choice of metrics from a large available set. In preliminary work, we created a domain-independent approach for evaluating foundation model outputs across text, audio, and images, with standardized support for visualization and automation of downstream processing, called GAICo (Generative AI Results Comparator). We will now specialize it to industry domains to meet the growing demand for reliable, domain-aware evaluation tools as generative AI becomes embedded in high-stakes environments. Its long-term impacts include reducing user risks, improving accountability in AI deployment, and establishing a new standard for responsible AI evaluation.

This I-Corps project uses experiential learning, coupled with first-hand investigation of the industry ecosystem, to assess the translation potential of Generative AI for high-impact application domains. GenAI evaluation methods often treat all deviations from a reference as equivalent, which can hide domain-critical errors while overemphasizing trivial differences. GAICo was created from the need to compare GenAI results while training FMs in a previous NSF initiative, and has been widely downloaded. The new technology extends GAICo to application domains, starting with health, and is generalizable to others. In health, focusing on brain science, our scope includes biosignals such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking (ET). GAICo-Health will provide detailed evaluation of GenAI outputs through both reference-based and reference-free visualizations, metrics, and insights. The new system compares generated AI results in a domain-aware manner and produces severity-aware scores, visual explanations, and action reports. The approach integrates domain ontology, severity-weighted scoring, stage-level error attribution, escalation estimation, domain-specific visualization, and action dispatch in one evaluation workflow.