Token Optimizer

Installation

Add the dependency to your pom.xml:

<dependency>
    <groupId>dev.sassine</groupId>
    <artifactId>token-optimizer</artifactId>
    <version>1.1.3</version>
</dependency>

Basic Usage

The simplest way to use Token Optimizer:

import dev.sassine.tokenoptimizer.TokenOptimizer;
import dev.sassine.tokenoptimizer.OptimizationResult;
import java.util.HashMap;
import java.util.Map;

// Create your data object
Map<String, Object> person = new HashMap<>();
person.put("name", "John");
person.put("age", 30);
person.put("city", "New York");

// Optimize - automatically chooses JSON or TOON
OptimizationResult result = TokenOptimizer.optimize(person);

// Use the optimal format
System.out.println("Optimal format: " + result.getOptimalFormat());
System.out.println("Optimal content:\n" + result.getOptimalContent());
System.out.println("Token savings: " + result.getTokenSavings() + " tokens");

Token Counting Methods

Token Optimizer supports two methods for counting tokens:

1. Generic Estimation (Default)

Fast approximation that works for any LLM model. Uses a combination of character-based and word-based estimation.

// Uses generic estimation
OptimizationResult result = TokenOptimizer.optimize(obj);

2. Model-Specific Tiktoken

Accurate token counting using the actual tokenizer for specific models (GPT-4, GPT-3.5, Claude, etc.). Requires the jtokkit library (already included as a dependency).

import com.knuddels.jtokkit.api.ModelType;

// Use GPT-4 tokenizer
OptimizationResult result = TokenOptimizer.optimize(obj, ModelType.GPT_4);

// Use GPT-3.5 Turbo
OptimizationResult result2 = TokenOptimizer.optimize(obj, ModelType.GPT_3_5_TURBO);

// Use Claude (uses GPT-4 encoding)
OptimizationResult result3 = TokenOptimizer.optimize(obj, ModelType.GPT_4);

Optimization Criteria

Choose what metric to optimize for: tokens (for LLMs), bytes (for storage), or characters (for size analysis). Different criteria can produce different optimal formats!

// Optimize by tokens (default - for LLM usage)
OptimizationResult byTokens = TokenOptimizer.optimize(person);

// Optimize by bytes (for data persistence/storage)
OptimizationResult byBytes = TokenOptimizer.optimizeByBytes(person);

// Optimize by characters (for size analysis)
OptimizationResult byChars = TokenOptimizer.optimizeByCharacters(person);

// From JSON string
OptimizationResult byBytes = TokenOptimizer.optimizeFromJsonByBytes(jsonString);
OptimizationResult byChars = TokenOptimizer.optimizeFromJsonByCharacters(jsonString);

Optimization Policy

Control how the optimizer selects formats using OptimizationPolicy. This allows you to set minimum savings thresholds or force a specific format.

import dev.sassine.tokenoptimizer.OptimizationPolicy;
import dev.sassine.tokenoptimizer.PayloadFormat;

// Create a policy with 5% minimum savings threshold
OptimizationPolicy policy = OptimizationPolicy.builder()
    .preferFormat(PayloadFormat.AUTO)              // AUTO, JSON_ONLY, or TOON_ONLY
    .minSavingsPercentForSwitch(5.0)               // Only switch if savings ≥ 5%
    .build();

// Use the policy
OptimizationResult result = TokenOptimizer.optimize(person, policy);

// Policy options:
// - PayloadFormat.AUTO: Choose format with lowest tokens (respects threshold)
// - PayloadFormat.JSON_ONLY: Always use JSON
// - PayloadFormat.TOON_ONLY: Always use TOON

Multiple Metrics

Get comprehensive metrics for both LLM usage (tokens) and data persistence (characters, bytes):

OptimizationResult result = TokenOptimizer.optimize(person);

// Token metrics (for LLM usage)
int tokens = result.getOptimalTokenCount();
double tokenSavings = result.getTokenSavingsPercentage();

// Character metrics (for size analysis)
int chars = result.getOptimalCharacterCount();
double charSavings = result.getCharacterSavingsPercentage();

// Byte metrics (for data persistence/storage)
int bytes = result.getOptimalByteCount();
double byteSavings = result.getByteSavingsPercentage();

// Print all metrics
System.out.println(result);
// Output: OptimizationResult{optimalFormat=TOON, tokens=110/977/110 (savings: 867, 88.74%), 
//          chars=450/3200/450 (savings: 2750, 85.94%), bytes=450/3200/450 (savings: 2750, 85.94%)}

Reverse Conversion (TOON → JSON/Object)

Convert TOON strings back to JSON or Java objects:

// TOON → JSON
String toonString = "name: John\nage: 30";
String json = TokenOptimizer.fromToonToJson(toonString);
// Result: {"name":"John","age":30}

// TOON → Object
Object obj = TokenOptimizer.fromToon(toonString);
// Result: Map with name="John", age=30

// TOON → Typed Class
MyClass obj = TokenOptimizer.fromToon(toonString, MyClass.class);

// Using ToonConverter directly
String json = ToonConverter.toJson(toonString);
Object obj = ToonConverter.fromToon(toonString);