TeachFX: Refuel helps TeachFX ship AI features in 2 weeks instead of 2 months

Executive Summary

TeachFX, an ed-tech company aiming to enhance classroom interaction, partnered with Refuel to evolve their product by introducing ML features for detecting key educational moments in classroom sessions. Refuel enabled TeachFX to quickly generate accurate training datasets, achieving over 92% agreement with expert annotators, and significantly reduced the time required for feature development from months to just two weeks. This collaboration supports TeachFX's mission to improve education outcomes by accelerating the development of their ML features, underscoring the potential of LLMs for data labeling efficiency and quality.

‍

About TeachFX

TeachFX is a Bay Area-based, ed-tech company with a mission to promote more meaningful and equitable classroom dialogue by superpowering teachers’ work — using technology to provide educators with regular, automated feedback on their practice.

‍

Challenges and objectives

TeachFX is in the process of evolving their product offering — going from a detailed review generated after every single classroom session to providing teachers an interactive experience that enables them to dive deep into critical learning moments from the classroom and connect them across other sessions they’ve had with students.

The TeachFX team identified several product features they’d like to build that would allow teachers to gain an in-depth understanding of each classroom session. For example, detecting when a teacher created an “opportunity to respond” - a learning moment that facilitated responses from students, or when there was a moment of “teacher feedback” - a moment when a teacher provided feedback to a student. Detecting such moments in a classroom session requires an extremely nuanced understanding of teacher-student interactions, which requires capable ML models, and in-turn, high-quality training datasets.

TeachFX was looking for a solution that could help them consistently create high-quality training datasets and fine-tune LLMs for their ML features, and so they turned to Refuel.

‍

How Refuel helped

The TeachFX team used Refuel to build two new ML features — detecting “opportunities to respond” and “teacher feedback”. For each of them, they followed a consistent process:

• TeachFX team ingests their unlabeled dataset of anonymized classroom recordings into Refuel. They write simple natural language instructions for the data labeling task to get the dataset creation started.
• Refuel surfaces low-confidence examples for the TeachFX team for feedback, and provides explanations for labels, which helps the TeachFX team iterate on instructions and label quality. Within a few hours of working within the Refuel product, the TeachFX team was able achieve 92%+ agreement with their expert annotators.
• Refuel was used to label 10,000 diverse examples to produce a high-quality training dataset. The labeling process took 2 hours, instead of the 4 weeks of human effort it would have taken otherwise.

‍

As Berk Coker, CTO of TeachFX says:

”In a world where we are fine-tuning LLMs, all we need to do is find some good examples. We have immense unlabeled data, and Refuel’s product can go find many other examples for us. When the examples we’re looking for are not common, finding them and hand labeling takes a lot of time. So just being able to use LLMs to find the data, train a model and then boom, we have the insight. So it is a much, much faster iteration.”

‍

Results

• The TeachFX team was able to create a training dataset where the agreement with expert labels was 92%. Using this dataset, their ML team was able to train a smaller, task-specific model they could deploy online to power their new ML feature.
• The data labeling for their dataset took 2 hours instead of 100+ hours of manual effort it would have taken otherwise.
• With the new product vision in mind, the TeachFX team needs to deliver on 10s of ML features on their roadmap. With Refuel as a partner, they have a consistent process for shipping new ML features where they’re able to go from start to finish in 2 weeks, instead of 2+ months previously.

‍

“Refuel drastically reduces time to market for our ML features. Our team used to spend so much time on getting our training datasets in order for every new feature, but with Refuel, we can dramatically accelerate our roadmap.”
‍
Berk Coker, CTO at TeachFX

‍

Future Plans

The TeachFX team has an aggressive product and ML roadmap to deliver to on their mission for teachers and improving education outcomes. Refuel will be a key partner for the TeachFX engineering team to improve data quality and fine-tune custom LLMs to support TeachFX’s inspiring mission.

If you’re interested in how LLMs can help speed up data labeling and improve data quality, sign up here to get a demo of Refuel: https://www.refuel.ai/get-started

‍