I had Gemini train its own replacement for $9

Your AI-powered text analysis can be significantly cheaper if you first label data with a powerful LLM and then fine-tune a smaller, open-source model to do the work locally.

Peter Vijeh details his experiment using Gemini Pro to label product entities in Reddit comments for a knife enthusiast forum, aiming to reduce costs associated with continuous LLM API calls. Initially, Gemini 3.1 Pro was used for Named Entity Recognition (NER) to extract knife brands, models, and steel types. However, the API costs grew with comment volume, prompting a shift to a cheaper, open-source model called GLiNER. While zero-shot GLiNER achieved a lower accuracy (0.65 F1), Vijeh trained it on labels generated by Gemini. He provided Gemini with 4,290 Reddit comments, from which it extracted product names for $9. This labeled data was then used to fine-tune GLiNER. Despite initial training failures due to configuration issues and an incorrect understanding of the `words_mask` tensor, Vijeh eventually succeeded. The fine-tuned GLiNER model achieved an F1 score of 0.83 against Gemini's labels on unseen comments, utilizing a local Tesla T4 GPU for processing. The total cost for labels was $9, with an additional $2.50 for GPU time.

This experiment demonstrates a viable and cost-effective strategy for implementing specialized natural language processing tasks. By leveraging a large language model for initial data labeling, even with its associated per-call costs, the output can be used to train a much more efficient, locally runnable model. This approach circumvents the ongoing expense of large models for repetitive tasks and highlights the power of fine-tuning smaller, domain-specific models for high accuracy.

The success hinges on the ability to generate high-quality training data from a powerful LLM and then successfully fine-tune a more constrained model. The debugging challenges encountered underscore the complexities of model training pipelines, where subtle issues with tensor handling can lead to significant development time. This method is applicable to any scenario requiring entity extraction or classification from large volumes of text where a per-instance LLM cost would be prohibitive.