OpenJev

You can now run and test large language models for decision making on your own hardware, which could lead to new privacy-preserving applications.

OpenJev, a new experimental project, allows users to run decision-modeling language models directly in their web browser without a backend. This experiment, available on GitHub, aims to demonstrate running models similar to Jev locally. Users can select from various model sizes, including Qwen3 0.6B for phones and MiniCPM5 2B for desktops, with a larger Qwen3.5 4B option available for high-memory devices. The project showcases two methods for extracting decisions: direct readout of probability logits and token-by-token generation of JSON probabilities. Both methods utilize the same local model, with timings measured for setup, model loading, warmup, and execution. Model weights are downloaded and cached in the browser, with inputs remaining on the page to ensure privacy. Performance varies based on model size and device capabilities, with all timings reflecting real-time execution rather than pre-recorded results. The demo uses quantized GGUF builds via wllama, noting that quantization can affect both speed and quality.

The OpenJev experiment signifies a growing trend toward on-device AI processing, enabling complex tasks like decision modeling without relying on external servers. This shift is crucial for applications where data privacy and low latency are paramount, potentially democratizing access to AI capabilities for a wider range of users and devices, including those with limited internet connectivity.

By enabling local execution of models like Jev, OpenJev challenges traditional cloud-based AI service models and highlights the potential for web browsers to become powerful platforms for advanced AI computation. The project's focus on transparent, real-time performance metrics for direct readout versus generative methods also provides valuable insights into the practical trade-offs of different model interaction techniques for developers and researchers.