SE Packet-Based NPUs In The LLM Era: From Compute-Bound CNNs To Memory-Bound Edge And Automotive Workloads

[admin]

Many Semiconductor Engineering readers know the basic story behind Expedera’s Origin NPU IP architecture: packets instead of layers, higher MAC utilization, and less gratuitous movement of activations to external memory. What’s changing now is the workload mix. Vision-only edge processors are giving way to systems where LLMs, VLMs and generative features sit alongside traditional perception networks in smartphones, vehicles and embedded gateways. In terms of NPU design, what are the implications when the dominant edge workloads stop being purely compute-bound CNNs and start looking like memory-bound transformers with large KV caches—especially in safety-critical automotive contexts? Workload inversion: CNNs vs LLMs/VLMs The first lens shift is recognizing the inversion in bottlenecks. For CNN-based vision workloads (YOLO-class detectors, MobileNet-style classifiers, segmentation networks), the dominant constraint has historically been compute. Weights are reused, the execution is largely stateless at inference time, and accelerators were designed around maximizing MAC throughput within a given area and power envelope. Expedera’s packet-based scheduling improved MAC utilization and reduced DDR traffic by distributing work across packets rather than monolithic layers. LLMs and VLMs change the story. As Expedera and others have been highlighting, prefill can still be heavy on arithmetic, but decode quickly becomes dominated by KV cache access and enlarged parameter sets with low effective reuse. The result is a workload that is increasingly memory-bound, not compute-bound, particularly when models are running in resource-constrained edge and in-vehicle environments. Fig. 1: Increased runtime inference complexity with LLMs. Moreover, CNN-centric NPUs were built for stateless, parallel, compute-bound inference, while VLM transformer decodes are now limited by memory behavior—KV cache size, access patterns, and bandwidth—not by the nominal TOPS rating on the datasheet. Packets in a memory-bound world Given this workload inversion, we looked at whether the packet abstraction that worked well for CNNs could be extended across transformer blocks and KV cache access, and co-designed hardware and software around LLM/VLM behavior rather than treating them as “just another model.” At a high level, the engine:

• Scales to 128 TFLOPS in a single core, and to PetaFLOPS with multi-core configurations, but does so in a way that keeps memory behavior central rather than incidental.
• Uses discrete processing blocks for feed-forward, attention and vector operations, with packets routed across these blocks according to the network’s structure and the current phase (prefill vs decode).
• Maintains the original promise: models run “as trained,” with no retraining and no accuracy reduction, while packetization and scheduling are managed within the hardware/software stack.

The critical data point for LLMs is memory movement. We found that, compared to alternative approaches, packet-based processing can reduce external memory moves by more than 75% for Llama 3.2 1B and Qwen2 1.5B—networks that are representative of real embedded LLM deployments rather than toy examples. Even in highly memory-bound scenarios, we’ve seen the architecture can deliver “thousands of effective TFLOPS” and “dozens of tokens per second per mm² of silicon,” pointing to a practical balance between compute and memory rather than an extreme in either direction. Edge and automotive implications: Latency, privacy, and power For edge and automotive systems, the packet story in LLM/VLM workloads translates directly into system-level constraints.

• Latency and privacy. Running LLM inference locally reduces end-to-end latency and keeps in-cabin or on-device data out of the cloud, which is increasingly a requirement for both user experience and regulatory reasons.
• Memory and power budgets. In vehicles, edge nodes and mobile devices, memory is heavily constrained. Streaming KV cache to DDR or HBM on every decode step quickly becomes a power and cost problem. Packet-based scheduling that reduces external memory moves and increases local reuse translates into more tokens per second within a given power budget.
• Heterogeneous workload coexistence. Automotive SoCs need to run perception, driver monitoring, infotainment, and now generative features together. An NPU family that treats CNNs and LLMs as first-class citizens—rather than bolting LLM support onto a CNN-optimized core—simplifies platform design and reduces the need for separate accelerators.

The recent “Best Edge AI Processor IP” award for Origin Evolution in the 2026 Edge AI and Vision Product of the Year Awards highlights the importance of a systems approach to solving the memory and power bottlenecks that prevent real-world deployment of GenAI at the edge, rather than just higher peak compute metrics. EDA and architecture: Modeling prefill and decode separately For EDA teams, the interesting part is less about packets as a concept and more about how to model and optimize prefill and decode phases explicitly. Origin Evolution treats prefill and decode differently:

• Prefill still wants high arithmetic throughput—matrix-multiply-heavy, closer to classic CNN behavior but with larger models.
• Decode is dominated by KV cache reads, memory streaming, and smaller per-step compute, potentially distributed across packets in a way that reduces external memory pressure and spreads work across attention and vector blocks.

Fig. 2: Inference event timeline comparison. Packet streams and discrete blocks make it possible to:

• Build performance and bandwidth models that differentiate between prefill and decode, rather than lumping transformer inference into a single average profile.
• Explore cache hierarchy and streaming interface parameters (DRAM/HBM width, SRAM size, partitioning) against real KV cache access patterns, tuned per phase.
• Analyze worst-case latency and bandwidth impact in safety-critical automotive workloads where LLM or VLM features are part of the human-machine interface or driver monitoring stack.

The packet abstraction remains the foundation, but system designers can think of packets as units of LLM/VLM behavior—scheduled across phase-specific blocks and cache hierarchies—rather than only as CNN layer fragments. Fig. 3: Packets – a unique approach to AI optimization. Out-of-the-box compatibility and software-first flows From a software and EDA tooling perspective, another lens is compatibility. Origin Evolution’s stack is built to ingest networks from HuggingFace, Llama.cpp, TVM and others, while supporting full integer and floating-point precisions, mixed modes, layer fusion/fission, and centralized control of multiple cores at the chip or chiplet level. The practical point is that packetization is not a new programming model for users. In practice, existing trained models can be dropped onto the hardware with no reduction in accuracy and no retraining requirements, and that packetization happens in the compiler/runtime path. For EDA and verification teams working in increasingly software-first procurement environments, this matters:

• It reduces the need for bespoke hardware-specific rewrites in the ML stack.
• It allows performance modeling and regression testing against reference implementations (e.g., Llama 3, Qwen 2) without rebuilding networks for each architecture.
• It aligns with the trend in automotive and mobile platforms where NPU IP is evaluated not just on microarchitecture, but on how fast and safely it can run real open-source models.

Looking ahead For edge AI designs, the open questions are less about whether packets work—they clearly do for CNNs—and more about how the architecture eases the complexity of modeling, scheduling, and verification across heterogeneous AI workloads. As GenAI features move into cars and edge devices at scale, those questions will likely determine which accelerator architectures actually make it into production and which remain confined to benchmark slides. The post Packet-Based NPUs In The LLM Era: From Compute-Bound CNNs To Memory-Bound Edge And Automotive Workloads appeared first on Semiconductor Engineering.

Source: https://semiengineering.com/packet-base ... workloads/