PowerInfer
High-speed large language model locally deployed inference engine
Product Details
PowerInfer is an engine for high-speed large language model inference on PCs using consumer GPUs. It exploits the high locality feature in LLM inference by preloading thermally activated neurons onto the GPU, thereby significantly reducing GPU memory requirements and CPU-GPU data transfer. PowerInfer also integrates adaptive predictors and neuron-aware sparsity operators to optimize the efficiency of neuron activation and computational sparsity. It can perform inference on a single NVIDIA RTX 4090 GPU at an average generation rate of 13.20 tokens per second, which is only 18% lower than the top server-grade A100 GPU. while maintaining model accuracy.
Main Features
Target Users
PowerInfer is suitable for high-speed inference of large language models deployed locally.
Quick Access
Visit Website →Categories
Related Recommendations
Discover more similar quality AI tools
Gitee AI
Gitee AI brings together the latest and hottest AI models, provides one-stop services for model experience, inference, training, deployment and application, provides abundant computing power, and is positioned as the best AI community in China.
MouSi
MouSi is a multi-modal visual language model designed to address current challenges faced by large-scale visual language models (VLMs). It uses integrated expert technology to collaborate the capabilities of individual visual encoders, including image-text matching, OCR, image segmentation, etc. This model introduces a fusion network to uniformly process outputs from different vision experts and bridge the gap between image encoders and pre-trained LLMs. In addition, MouSi also explored different position encoding schemes to effectively solve the problems of position encoding waste and length limitation. Experimental results show that VLMs with multiple experts exhibit superior performance than isolated visual encoders, and obtain significant performance improvements as more experts are integrated.
OpenAI Embedding Models
OpenAI Embedding Models is a series of new embedding models, including two new embedding models and updated GPT-4 Turbo preview models, GPT-3.5 Turbo models, and text content review models. By default, data sent to the OpenAI API is not used to train or improve OpenAI models. New embedding models with lower pricing include the smaller, more efficient text-embedding-3-small model and the larger, more powerful text-embedding-3-large model. An embedding is a sequence of numbers that represents a concept in something like natural language or code. Embeddings make it easier for machine learning models and other algorithms to understand the relationships between content and perform tasks such as clustering or retrieval. They provide support for knowledge retrieval in the ChatGPT and Assistants APIs, as well as many retrieval augmentation generation (RAG) development tools. text-embedding-3-small is a new efficient embedding model. Compared with its predecessor text-embedding-ada-002 model, it has stronger performance. The average MIRACL score increased from 31.4% to 44.0%, while the average score in the English task (MTEB) increased from 61.0% to 62.3%. Pricing for text-embedding-3-small is also 5x lower than the previous text-embedding-ada-002 model, from $0.0001 per thousand tags to $0.00002. text-embedding-3-large is a new generation of larger embedding models, capable of creating embeddings of up to 3072 dimensions. With stronger performance, the average MIRACL score increased from 31.4% to 54.9%, while the average score in MTEB increased from 61.0% to 64.6%. text-embedding-3-large is priced at $0.00013/thousand marks. Additionally, we support native functionality for shortening embeddings, allowing developers to trade off performance and cost.
Adept Fuyu-Heavy
Adept Fuyu-Heavy is a new multi-modal model designed specifically for digital agencies. It performs well in multimodal reasoning, particularly in UI understanding, while also performing well on traditional multimodal benchmarks. Furthermore, it demonstrates our ability to extend the Fuyu architecture and obtain all associated benefits, including processing images of arbitrary sizes/shapes and efficiently reusing existing transformer optimizations. It also has the ability to match or exceed the performance of models of the same computational level, albeit requiring some of the capacity to be devoted to image modeling.
Meta-Prompting
Meta-Prompting is an effective scaffolding technique designed to enhance the functionality of language models (LM). This method transforms a single LM into a multi-faceted commander, adept at managing and integrating multiple independent LM queries. By using high-level instructions, meta-cues guide LM to decompose complex tasks into smaller, more manageable subtasks. These subtasks are then handled by different "expert" instances of the same LM, each operating according to specific customized instructions. At the heart of this process is the LM itself, which, as the conductor, ensures seamless communication and effective integration between the outputs of these expert models. It also leverages its inherent critical thinking and robust validation processes to refine and validate the final results. This collaborative prompting approach enables a single LM to simultaneously act as a comprehensive commander and a diverse team of experts, significantly improving its performance in a variety of tasks. The zero-shot, task-agnostic nature of meta-cues greatly simplifies user interaction, eliminating the need for detailed task-specific instructions. Furthermore, our research shows that external tools, such as the Python interpreter, can be seamlessly integrated with the meta-hint framework, thereby broadening its applicability and utility. Through rigorous experiments with GPT-4, we demonstrate that meta-cueing outperforms traditional scaffolding methods: averaged across all tasks, including the 24-point game, One Move General, and Python programming puzzles, meta-cueing using the Python interpreter feature outperforms standard prompts by 17.1%, is 17.3% better than expert (dynamic) prompts, and is 15.2% better than multi-personality prompts.
WARM
WARM is a solution for aligning large language models (LLMs) with human preferences through the Weighted Average Reward Model (WARM). First, WARM fine-tunes multiple reward models and then averages them in the weight space. Through weighted averaging, WARM improves efficiency compared to traditional predictive ensemble methods, while improving reliability under distribution shifts and preference inconsistencies. Our experiments show that WARM outperforms traditional methods on summarization tasks, and using optimal N and RL methods, WARM improves the overall quality and alignment of LLM predictions.
ReFT
ReFT is a simple and effective way to enhance the inference capabilities of large language models (LLMs). It first warms up the model through supervised fine-tuning (SFT), and then uses online reinforcement learning, specifically the PPO algorithm in this article, to further fine-tune the model. ReFT significantly outperforms SFT by automatically sampling a large number of reasoning paths for a given problem and naturally deriving rewards from real answers. The performance of ReFT may be further improved by incorporating inference-time strategies such as majority voting and re-ranking. It is important to note that ReFT improves by learning the same training problem as SFT without relying on additional or enhanced training problems. This shows that ReFT has stronger generalization ability.
Contrastive Preference Optimization
Contrastive Preference Optimization is an innovative approach to machine translation that significantly improves the performance of ALMA models by training the model to avoid generating translations that are merely adequate but not perfect. This method can meet or exceed the performance of WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.
GLM-4
Zhipu AI released GLM-4 and CogView3 at the first Technology Open Day. The overall performance of GLM-4 has been improved by nearly 60%, supporting longer context, stronger multi-modal support and faster reasoning. CogView3 approaches the multi-modal generation capabilities of DALL·E 3. The product is positioned as the next generation of base model and image generation AI.
Chain-of-Table
Chain-of-Table is a reasoning linked list framework for table understanding, specially designed to handle tasks such as table-based question answering and fact verification. It uses tabular data as part of the reasoning chain and guides large language models to perform operation generation and table updating by learning in context, thereby forming a continuous reasoning chain that demonstrates the reasoning process for a given tabular problem. This chain of reasoning contains structured information about intermediate results, enabling more accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art performance on multiple benchmarks including WikiTQ, FeTaQA and TabFact.
DocGraphLM
DocGraphLM is a document graph language model for information extraction and question answering. It uses advanced visual rich document understanding technology, combining pre-trained language models and graph semantics. It is unique in that it proposes a joint encoder architecture to represent documents and adopts a novel link prediction method to reconstruct the document graph. DocGraphLM predicts the direction and distance between nodes via a convergent joint loss function, prioritizing neighborhood recovery and downweighting remote node detection. Experiments on three SotA datasets show that employing graphical features can achieve consistent improvements in information extraction and question answering tasks. Furthermore, we report that employing graphical features accelerates convergence during training, even though these features are only constructed via link prediction.
TinyGPT-V
TinyGPT-V is an efficient multi-modal large-scale language model implemented by using a small backbone network. It has powerful language understanding and generation capabilities and is suitable for various natural language processing tasks. TinyGPT-V uses Phi-2 as a pre-trained model, which has excellent performance and efficiency.
Gemini AI
Gemini is Google's most powerful and versatile AI model, designed to be multi-modal and optimized for three different sizes: Ultra, Pro, and Nano. Gemini models offer superior performance and next-generation features to provide powerful AI support for a variety of applications. It provides scalable, efficient solutions with a focus on responsibility and security. Gemini models are already available on the market.
LEO
LEO is a multi-modal, multi-task omnipotent agent based on a large language model, capable of sensing, localizing, reasoning, planning and executing tasks in the 3D world. LEO is implemented through two stages of training: (i) 3D visual language alignment and (ii) 3D visual language action command adjustment. We carefully curate and generate a large-scale dataset containing object-level and scene-level multi-modal tasks that require deep understanding and interaction with the 3D world. Through rigorous experiments, we demonstrate LEO's outstanding performance on a wide range of tasks including 3D subtitles, question answering, reasoning, navigation, and robot operation.
AndesGPT
AndesGPT Andes large model is a personalized and exclusive large model and agent released by OPPO. It is based on the device-cloud collaboration architecture design and provides a variety of model specifications with different parameter sizes. It supports technical features such as dialogue enhancement, personalization and device-cloud collaboration. OPPO will lay out large-scale forward-looking technologies, cooperate with the Intelligent Computing Joint Laboratory established by the University of Science and Technology of China, and open source the agent framework to support the efficient incubation, hosting and application of agents.
Zidong Taichu
Zidong Taichu is a new generation large model launched by the Institute of Automation of the Chinese Academy of Sciences and the Wuhan Institute of Artificial Intelligence. It supports comprehensive question and answer tasks such as multiple rounds of question and answer, text creation, image generation, 3D understanding, signal analysis, etc., and has stronger cognitive, understanding, and creative capabilities. It has a wide range of application scenarios, including text creation, knowledge question and answer, image, text and sound understanding, music generation, 3D understanding and signal analysis and other functions. The product is positioned to provide high-quality artificial intelligence interactive experience.