tiktoken

Notion RAG Cost Management System: Built a comprehensive cost tracking infrastructure for the Notion RAG CLI using tiktoken to provide complete transparency into Gemini API usage. Implemented pre-call token estimation that calculates expected costs before sending requests, allowing users to understand cost implications of their queries. Created persistent logging to cost_log.json that records detailed breakdowns of input tokens (context + prompt) and output tokens (generated responses) for every query, with timestamps and cost calculations based on Gemini's pricing model. Users can query their cumulative costs mid-session using the costs command, which displays total expenditure, number of API calls, and average cost per query, enabling informed usage decisions and budget management.

Transparent Token Analytics: Designed the cost tracking system to give users deep visibility into exactly what LLMs consume during operation. For each query, the system displays not just the final cost, but the specific token counts for input (retrieved context + user question) and output (model response), helping users understand how different query types and context sizes impact API costs. This transparency proved invaluable for optimizing the RAG system's retrieval settings, as it revealed that retrieving 3-5 highly relevant chunks was more cost-effective than retrieving 10+ chunks of varying relevance, while maintaining answer quality.