In the accelerating race for faster artificial intelligence inference, while some companies like Cerebras are gaining traction with purpose-built chips, French startup Kog is taking a different approach. The company is betting on unlocking significantly more power from conventional GPUs already prevalent in data centers, such as AMD MI300X and Nvidia H200 units, through advanced software optimization.

Kog garnered significant attention with a tech preview demonstrating that “extremely fast single-request decoding is possible” on standard enterprise GPUs. This promise is particularly appealing as inference speed and associated costs have become critical bottlenecks for many AI applications. Businesses relying on AI workflows for professional tasks, including users of services like Claude Code who often face lengthy waits, stand to benefit. Even Anthropic, the developer of Claude, acknowledges the value of speed by offering a premium-priced “Fast Mode.” Kog aims to target customers deterred by these delays, including design partners who generate games and applications, for whom faster outcomes directly translate to increased revenue.

While the market for fine-tuning smaller models is still maturing, Kog has shifted its focus to accelerating the development of larger models to meet observed demand. The startup aims to deliver on its ambitious promise of “30x faster LLM inference.” Its initial demo showcased an impressive 3,000 per-request tokens per second (TPS) using a purpose-built small model, the open-sourced Laneformer 2B. CEO Gaël Delalleau remains confident that this deep-level optimization approach can be successfully applied to large language models (LLMs), asserting that modern GPUs, with their increasing memory bandwidth, are well-suited for decoding when properly optimized.

Delalleau’s unique background, combining solid-state physics with offensive cybersecurity, underpins Kog’s methodology. This expertise fosters a mindset of reverse-engineering hardware at a very low level to maximize its potential beyond its original design. This hands-on approach, however, is time-consuming, requiring weeks or months of dedicated research for each new GPU. With an 11-person team, this limits the number of chips Kog can support in the near term. Looking ahead, Kog plans to integrate its methodology into agent-based pipelines to scale support for more chips and models, aligning with Europe’s broader efforts to build indigenous AI capabilities. The immediate goal is to demonstrate 10x speed improvements on a major LLM by September, a crucial step for securing Series A funding and proving customer traction.

Key Takeaways

• Kog, a French startup, is focused on significantly boosting AI inference speeds on conventional GPUs through deep software optimization, rather than relying on new specialized hardware.
• The company's unique approach, rooted in solid-state physics and cybersecurity, aims to unlock up to 30x faster LLM inference, addressing critical bottlenecks in AI workflows for professional users.
• Kog's immediate strategy involves proving its optimization capabilities on major large language models by September to secure Series A funding and demonstrate market viability.

Editor’s Analysis & Impact

Kog’s innovative approach to AI inference could significantly disrupt the market by extending the lifecycle and capabilities of existing GPU infrastructure. This offers a compelling, cost-effective alternative for businesses seeking faster AI processing without the immediate need for substantial investments in specialized AI chips. If Kog successfully scales its deep-level optimization to major LLMs, it could democratize access to high-speed AI inference, accelerating AI adoption across various industries.

This development also highlights the increasing importance of software optimization in maximizing hardware potential, a trend that could reshape how companies approach AI infrastructure. Furthermore, Kog’s backing by European initiatives aligns with broader efforts to foster indigenous AI capabilities, potentially reducing reliance on external hardware solutions. The primary challenge for Kog will be scaling its labor-intensive, hands-on optimization process across a diverse and rapidly evolving landscape of GPU architectures.

Frequently Asked Questions

Q: What problem is Kog trying to solve in AI inference?
A: Kog aims to address the critical bottlenecks of AI inference speed and cost by significantly improving the performance of conventional GPUs through advanced software optimization. This allows businesses to achieve faster AI processing on existing hardware, making AI workflows more efficient and accessible.

Q: How does Kog achieve faster AI inference on existing GPUs?
A: Kog employs a deep-level software optimization strategy, leveraging insights from solid-state physics and cybersecurity. This involves reverse-engineering GPUs at a fundamental level to unlock their full potential for AI decoding, rather than relying solely on new hardware designs.

Q: What are Kog's immediate goals for its technology?
A: Kog's immediate focus is to demonstrate substantial speed improvements, specifically aiming for 10x faster inference, on major large language models by September. This achievement is crucial for securing its Series A funding round and proving the market traction and effectiveness of its software-driven approach.