Dongcha Beating AI News, Prime Intellect releases Prime Inference, an open source model inference service.
Users can directly call the model on demand; if the task requires long-term stable operation, a part of the inference computing power can also be locked in advance. The interface is compatible with the OpenAI API, and the first publicly deployed model is GLM-5.3.
This system has previously been used within Prime Intellect. Officially, it processes nearly 1 trillion tokens every day for reinforcement learning, synthetic data, model evaluation, and coding agents. Prime Intellect has also been running large-scale deployments for customers since January this year.
Prime Inference focuses on optimizing the long context load of Agent. It distributes prompt word processing and response generation on different GPUs, reducing the p90 inter-token latency by nearly 40% in the test. Prime Intellect also compresses the KV cache. Under the same video memory, the number of tokens that can be cached by each decoder increases from 1.09 million to 1.63 million, an increase of about 50%.
Prime Intellect has previously provided training infrastructure such as reinforcement learning, evaluation and sandboxing. After adding the inference service, it can connect model training and actual deployment to the same platform, and the data generated by actual operation can continue to be used for subsequent training.