Title:
AI Assist: ARP AI Pilot Project for Rating Open Responses in Casper
Agreement Number:
1028955
Agreement Value:
$75,000.00
Agreement Date:
Apr 1, 2025 - Mar 31, 2026
Description:
We propose a new approach to automated test scoring that combines recent advances in large language models (LLMs) with traditional clear-box ML approaches. We will use an LLM-as-a-judge framework in association with a rubric of scoring elements (developed in consultation with subject matter experts) to classify responses on the basis of these scoring elements. The LLM will also provide its reasoning for these decisions. We will then construct clear-box ML models that combine these scoring elements extracted from the responses into a single numeric score. This method will allow us to not only score a response, but also to identify precisely how individual elements of the rubric contributed to the score and why each element of the rubric was.
Organization:
National Research Council Canada
Expected Results:
The NRC IRAP Artificial Intelligence (AI) Assist Program aims to equip and support SMEs to incorporate AI into their core products and services and/or undertake scientific research, product development, testing, and validation. AI Assist aims to increase the number, the quality, and the competitiveness of Generative AI (Gen AI), Deep Neural Nets (DNN), and Deep Learning (DL) products created and delivered by SMEs.
Location:
Toronto, Ontario, CA M5V 2Y1
Reference Number:
172-2025-2026-Q1-1028955
Agreement Type:
Contribution
Report Type:
Grants and Contributions
Recipient Business Number:
836700997
Recipient Type:
For-profit organization
Recipient's Operating Name:
Acuity Insights Inc.
Recipient's Legal Name:
Acuity Insights Inc.
Federal Riding Name:
Spadina--Fort York
Federal Riding Number:
35101
Program:
Industrial Research Assistance Program – AI Assist
Program Purpose:
The NRC IRAP Artificial Intelligence (AI) Assist Program aims to equip and support SMEs to incorporate AI into their core products and services and/or undertake scientific research, product development, testing, and validation. AI Assist aims to increase the number, the quality, and the competitiveness of Generative AI (Gen AI), Deep Neural Nets (DNN), and Deep Learning (DL) products created and delivered by SMEs.
Coverage:
The AI Assist program is designed to help innovative SMEs navigate the challenges of developing and adapting generative AI (GenAI) and deep learning (DL) solutions and to assist SMEs with awareness, planning and execution in order to develop these technologies safely and ethically.