Mach X Runs LLMs on Arc Pro B70 Cards: 16 Agents at Once [2026]

By Indie Kings | October 2, 2026

Updated October 2, 2026: Mach X is a C++ and SYCL inference engine built for Arc and tuned on two Arc Pro B70 32GB cards per the Red-Weasel/machx-inference-engine GitHub README. It runs up to 16 parallel requests with agent swarms to 16 and shared prefix caching through an OpenAI compatible server per the same README.

Image: Mach X serving art. Credit: Red-Weasel via GitHub.

What Mach X Is

Mach X is an inference engine written in C++ with SYCL and built from the metal up for Intel Arc, per the Red-Weasel/machx-inference-engine GitHub README. It is not a llama.cpp fork and it does not depend on PyTorch or a vendor runtime, per the same README. That design choice matters for Arc owners because the code path targets Intel GPUs directly instead of passing through a generic layer.

The project reports Apache 2.0 licensing with 4 stars and 19 commits, per the repository profile noted in the assignment brief. Those numbers describe a small focused project rather than a large foundation backed stack. Readers should treat it as an early hardware tuned effort with active tuning but narrow hardware coverage. Check the live commit history before planning production use.

Kernel work is hand written SYCL with XMX acceleration, int dot GEMV paths, and tiled FlashAttention, per the Red-Weasel/machx-inference-engine GitHub README. Those three pieces cover matrix math for small batch inference, quantized dot products, and attention tiling for longer context. The README presents them as Arc specific tuning rather than portable generic kernels. Exact kernel coverage per model needs a fresh README check [NEEDS VERIFICATION for per model kernel mapping].

The server speaks an OpenAI compatible protocol with tool calls, per the Red-Weasel/machx-inference-engine GitHub README. That means scripts written for an OpenAI style endpoint can point at a local address with limited changes in many cases. Tool call support includes structured tool calls plus reasoning_content fields, per the same README. Client compatibility varies by SDK version so test your client build [NEEDS VERIFICATION for specific SDK versions].

Multi GPU support uses tensor parallel execution across two cards, per the Red-Weasel/machx-inference-engine GitHub README. The tuned reference is two Arc Pro B70 32GB cards for a combined 64GB pool. Tensor parallel mode splits weight work across both GPUs during serving. Single card behavior is not the tuned focus so expect manual tuning outside the reference setup [NEEDS VERIFICATION for single card guidance].

Auto lane counts select concurrency settings at serve time, per the Red-Weasel/machx-inference-engine GitHub README. Lanes here mean parallel request slots that share the loaded model. The server can open up to 16 parallel requests in the reference config, per the same README. This design favors agent swarms and batch testing over single chat use.

Agent swarms scale to 16 concurrent agents with shared prefix caching, per the Red-Weasel/machx-inference-engine GitHub README. Shared prefix caching reuses prompt prefix work across requests with the same system prompt or document head. That reuse cuts repeated work when 16 agents share instructions. Hit rates depend on prompt design so structure prompts to share prefixes where possible.

Quant support includes native Q6_K and Q5_K formats, per the Red-Weasel/machx-inference-engine GitHub README. Native support means the engine reads those quant types directly without an external conversion step in the documented path. Q6_K keeps more weight fidelity than Q5_K at the cost of larger files in general quant practice. Exact size and quality tradeoffs per tuned model need a README check [NEEDS VERIFICATION for per model quant files].

Stopping behavior is described as fast and safe, per the Red-Weasel/machx-inference-engine GitHub README. Fast safe stops matter when 16 lanes run at once because one stuck request should not block the rest. The server path handles stop requests, tool call boundaries, and reasoning field handling in one place. Stress test stops with your own swarm script before long runs [NEEDS VERIFICATION for stop latency numbers].

For Indie Kings readers the key point is focus. This is not a general CUDA server ported to Arc as an afterthought. It is a small C++ and SYCL codebase tuned on two B70 cards with hand kernels, tensor parallel multi GPU, auto lanes, 16 parallel requests, structured tool calls, reasoning_content fields, native Q6_K and Q5_K, and fast safe stops, per the Red-Weasel/machx-inference-engine GitHub README. That focus brings speed on the reference hardware but narrow portability. Owners of the exact two card setup gain the most.

Search intent in 2026 for phrases like run LLM on Arc, Arc Pro B70 inference, SYCL LLM server, and multi GPU local agents points to one need [NEEDS VERIFICATION for search volume]. Readers want a stable recipe that uses both B70 cards at once without cloud billing. Mach X answers that intent on paper with OpenAI compatibility and swarm support. This guide shows how to confirm hardware, install, load a model, serve with auto lanes, and run agents while reading metrics with care.

Tuned Models

Mach X lists seven tuned models with dated benchmark rows on two B70 cards, per the Red-Weasel/machx-inference-engine GitHub README. The families named in the brief are MiMo V2.6 Flash, DeepSeek V4.1 and V4 Flash, GLM 5.3 Flash, Qwen3.8 Flash, Qwen3.6 35B A3B class, and Qwen3.8 27B. All figures below are repo reported with dates as shown in the README and are never our measurements. Exact rates change with build flags and driver versions so recheck the live README before quoting a number.

MiMo V2.6 Flash is listed as a tuned low latency option for the two card setup, per the Red-Weasel/machx-inference-engine GitHub README. The README carries a dated benchmark row for this model on two B70 cards [NEEDS VERIFICATION for exact date and rate]. Lane caps apply to MiMo in the documented config, per the same README. Respect the cap rather than forcing 16 lanes on every model.

DeepSeek V4.1 and V4 Flash cover the reasoning heavy side of the tuned list, per the Red-Weasel/machx-inference-engine GitHub README. Both models have dated benchmark rows on two B70 cards in the README [NEEDS VERIFICATION for exact dates and rates]. DeepSeek greedy decoding can show variance across runs, per the same README. Use fixed seeds and repeated runs when comparing outputs.

GLM 5.3 Flash and Qwen3.8 Flash round out the Flash tier for fast responses, per the Red-Weasel/machx-inference-engine GitHub README. Each has a dated benchmark row on two B70 cards in the README [NEEDS VERIFICATION for exact dates and rates]. Flash naming here signals tuning for speed rather than maximum parameter count. Test prompt style matters because short instructions favor prefix cache hits.

Qwen3.6 35B A3B class and Qwen3.8 27B cover the larger end of the tuned set, per the Red-Weasel/machx-inference-engine GitHub README. Both carry dated benchmark rows on two B70 cards [NEEDS VERIFICATION for exact dates and rates]. Scheduling notes in the README single out the 35B class for special handling, per the same README. Read that scheduling section before running wide swarms on the 35B class model.

The table below summarizes five representative rows from the seven model set to keep the page readable. It preserves the repo reported framing and flags exact numbers for a live check. Use it as an index into the README rather than as a benchmark source. Always open the linked README row for the full context of batch size, quant, and date.

One token figures are the default unit in this small project unless a row says otherwise, per the brief context from the README. That means short single token tests may not predict long output speed. Long context, tool call JSON, and reasoning fields all add work beyond a single token probe. Time your own prompt shape with a fixed script.

Version context matters because tuning lands over time. The brief cites v0.2.6 agent swarm replay results as one dated data point, per the Red-Weasel/machx-inference-engine GitHub README and releases. Newer tags may retune lanes, prefix caching, or model defaults. Record the exact tag you run alongside every timing table you keep.

Model file handling stays local on your two card host in the documented path. Download weights once, verify checksums where the README provides them, and keep only the quants you test [NEEDS VERIFICATION for checksum availability]. Disk use grows fast with Q6_K files across seven families. Keep a simple log of model name, quant, tag, and date tested.

Choice guidance is simple for first runs. Start with a Flash tier model to validate lanes and prefix caching, then move to DeepSeek V4.1 for reasoning tests, then try the 35B class once scheduling is understood, per the general README order [NEEDS VERIFICATION for recommended order]. Change only one variable at a time between runs. That habit makes the dated README rows easier to interpret against your own logs.

Steps

These five steps follow the README order for a two card B70 host. Confirm hardware first, then install the engine, load a tuned model, serve with auto lanes, and run agents while reading metrics. Do them in order and log each result. Skipping hardware checks is the fastest way to waste a download.

1. Step 1: Confirm two B70 cards. Confirm two Arc Pro B70 32GB cards are seated, powered, and visible to the OS and Intel tools, per the Red-Weasel/machx-inference-engine GitHub README reference setup. Record driver version, kernel version, and device indices before installing anything [NEEDS VERIFICATION for exact driver baseline]. Check that both cards show 32GB each and that no card reports errors. Reboot once after driver changes so the check reflects a clean state.
   
   Log cooling and power at the same time. Confirm case airflow is clear, fans respond under load, and the power supply meets the two card demand [NEEDS VERIFICATION for wattage guidance]. A psykologically steady host matters here only in the plain sense that a cool and stable PC gives cleaner test numbers. Note idle temps and load noise in your runbook for later comparison.
   
   Save a hardware baseline file with board model, CPU, RAM, storage free space, OS build, driver version, and both GPU IDs. Take screenshots of device listings for support use. This file proves what changed when a later update shifts performance. Keep it next to your model notes.
2. Step 2: Install the engine. Clone or download the Red-Weasel/machx-inference-engine repository and follow the build steps in the GitHub README for Intel Arc. The codebase is C++ with SYCL and hand kernels using XMX, int dot GEMV, and tiled FlashAttention, per the same README. It avoids llama.cpp fork code, PyTorch, and vendor runtimes in the documented path, per the same README. Match compiler and SYCL toolchain versions to the README tag you plan to run [NEEDS VERIFICATION for exact toolchain versions].
   
   Build in a clean directory and keep the tag fixed during tests. Note Apache 2.0 licensing terms as you redistribute or modify code, per the repository profile. Small projects move fast so record the exact commit hash alongside v0.2.6 or newer tags. That hash lets you reproduce a result after later commits land.
   
   Expect a first request kernel compile cost after install, per the Red-Weasel/machx-inference-engine GitHub README. The first run builds tuned kernels before tokens flow, which makes the first timing slower than steady state. Warm up with a short prompt and discard the compile window from comparisons. Log warm versus steady numbers separately.
3. Step 3: Load a tuned model. Pick one tuned model from the seven family list, starting with a Flash tier option for the first load. Follow the README model setup for weights, quant choice, and config keys, choosing native Q6_K or Q5_K as documented, per the Red-Weasel/machx-inference-engine GitHub README. Verify file paths and quant labels before serving [NEEDS VERIFICATION for exact file names]. Keep the model tag and quant in your notes from the first load.
   
   Respect model specific lane caps at load time. MiMo and Flash models carry lane caps in the documented config, per the same README. Do not force 16 lanes on a capped model during the first test. Confirm the effective lane count in logs after load.
   
   Check tool call and reasoning field settings while the model loads. Structured tool calls and reasoning_content fields are part of the OpenAI compatible server, per the same README. Enable only the fields your client parses. Test with a short tool call prompt and inspect the JSON shape before wide runs.
4. Step 4: Serve with auto lanes. Start the OpenAI compatible server with tensor parallel multi GPU mode across both B70 cards, per the Red-Weasel/machx-inference-engine GitHub README. Enable auto lane counts so the server sizes parallel slots for the loaded model. Confirm the listening address and model name in startup logs. Test with one request before opening parallel lanes [NEEDS VERIFICATION for exact flags and ports].
   
   Verify lane behavior with a small sweep from one to the model cap. Watch for the MiMo and Flash lane caps noted in the README. Note where throughput stops scaling and where latency rises. Keep the sweep script fixed so later tags can be compared fairly.
   
   Confirm fast safe stops during this serve test. Send a stop mid stream and check that other lanes keep flowing, per the fast safe stop design in the same README. Tool call edge cases can appear at stop boundaries, per the same README. Log any malformed tool JSON with the prompt that caused it.
5. Step 5: Run agents and read metrics. Launch an agent swarm up to 16 with a shared system prefix to benefit from shared prefix caching, per the Red-Weasel/machx-inference-engine GitHub README. Use the v0.2.6 swarm replay pattern of many requests in one window as your template. The repo reported replay is 75 requests in 139.7 seconds, per the same README and releases, which frames expectations for swarm scale. Keep prompts structurally shared at the head to raise cache hits.
   
   Read metrics with the honest limits in mind. Single request versus shared prefix byte differences change what counts as input, per the same README. Scheduling notes apply to the 35B class only, per the same README. DeepSeek greedy variance and first request kernel compile cost also shift numbers, per the same README. Separate warmup, steady state, and swarm windows in your log.
   
   Save every run with tag, model, quant, lane count, prefix design, request count, total seconds, and error notes. The small project default is one token figures unless a row says otherwise, so note your prompt length and output length every time. That log turns a single swarm demo into repeatable evidence.

After the five steps, keep a one page runbook with hardware IDs, build hash, model and quant, serve flags, lane count, and swarm script path. That page lets you restart in minutes after a reboot. Update it whenever the README tag changes. Steady notes beat fast clicks for multi GPU work.

Agent Swarms and Prefix Cache

Agent swarms to 16 are a headline feature because they match real coding and research workflows, per the Red-Weasel/machx-inference-engine GitHub README. Sixteen parallel requests let planning, retrieval, coding, and review agents run at once against one local server. Tensor parallel multi GPU mode keeps both B70 cards busy during the swarm. Auto lane counts size the slots so operators do not hand tune every model.

Shared prefix caching is the efficiency core for swarms, per the Red-Weasel/machx-inference-engine GitHub README. When agents share the same system prompt, tool definitions, or document head, the server reuses that prefix work. Reuse cuts repeated compute and steadies tail latency under load. Design prompts with a stable head and variable tail to maximize hits.

The v0.2.6 agent swarm replay gives a concrete scale reference of 75 requests in 139.7 seconds, per the Red-Weasel/machx-inference-engine GitHub README and releases. That window averages about 1.86 seconds per request slot when divided evenly, though parallel lanes overlap in practice. Treat that math as a rough scale marker rather than a latency promise. Your prefix hit rate, output length, and lane count will move the result.

Prompt design decides cache value more than any flag. Put role instructions, tool schemas, and stable documents first, then agent specific tasks last. Keep formatting identical across agents so prefixes match byte for byte where possible. Note that single request versus shared prefix byte differences affect accounting, per the same README. Log both raw prompt bytes and cached prefix bytes when the server exposes them [NEEDS VERIFICATION for exact metric keys].

Structured tool calls need swarm level testing because edge cases grow with concurrency, per the Red-Weasel/machx-inference-engine GitHub README. Run a replay where several agents call tools at once and validate every JSON payload. Watch stop boundaries where truncated tool JSON can appear. Keep a corpus of failing prompts with the tag and model noted.

Reasoning_content fields add useful traces for planning agents, per the Red-Weasel/machx-inference-engine GitHub README. They also add output bytes that affect swarm time. Test with reasoning on and off using the same task set. Compare total seconds, output length, and task success rather than single token speed alone.

Start swarm tuning with a fixed 75 request script modeled on the v0.2.6 replay shape. Run it once with a shared prefix and once with divergent prefixes to show cache value. Then sweep lanes from 4 to 8 to 16 where model caps allow. Record total seconds for each shape with the same model, quant, and tag. That small matrix tells you more than any single headline number.

Limits

Honest limits keep multi GPU results comparable across runs. The README documents several behaviors that shift numbers if ignored, per the Red-Weasel/machx-inference-engine GitHub README. Read this section before quoting any swarm time. Each limit below comes from the repo rather than from our test bench.

Single request versus shared prefix byte differences change input accounting, per the Red-Weasel/machx-inference-engine GitHub README. A prompt that shares a cached prefix reports different byte counts than the same prompt run alone. Compare only runs with the same prefix shape. Log cache hit state alongside raw timing.

Scheduling notes apply to the 35B class only, per the Red-Weasel/machx-inference-engine GitHub README. The Qwen3.6 35B A3B class model follows scheduling rules that do not generalize to smaller Flash models. Do not copy 35B lane or batch settings onto MiMo or Flash models. Keep a separate config section for the 35B class in your runbook.

MiMo and Flash lane caps limit concurrency per model, per the Red-Weasel/machx-inference-engine GitHub README. Forcing extra lanes beyond the cap can hurt rather than help. Confirm the effective lane count in server logs after each config change. Treat the cap as part of the model definition, not as a suggestion.

Tool call edge cases remain in the documented path, per the Red-Weasel/machx-inference-engine GitHub README. Concurrent tool calls can expose malformed JSON at boundaries or stops. Validate tool payloads in code rather than trusting raw strings. File issues with prompt shape, tag, model, and lane count when you hit a repeatable case.

DeepSeek greedy variance means repeated runs can differ even with greedy settings, per the Red-Weasel/machx-inference-engine GitHub README. Use fixed seeds where supported and run repeats before claiming a delta [NEEDS VERIFICATION for seed flag support]. Report ranges rather than single best runs. That habit keeps reasoning model comparisons fair.

First request kernel compile cost slows the first timing window, per the Red-Weasel/machx-inference-engine GitHub README. SYCL kernels compile on first use, then steady state runs faster. Warm up the server with a short prompt and exclude compile time from swarm math. Log warm and steady numbers in separate columns.

The small project scope frames all of the above. With Apache 2.0 licensing, 4 stars, and 19 commits reported in the brief, coverage is narrow and docs can lag code. One token figures are the default unless a row notes longer outputs. Test your prompt length, keep tags pinned, and recheck the README before publishing numbers.

FAQ

What is Mach X in one sentence?
Mach X is a C++ and SYCL inference engine built for Intel Arc and tuned on two Arc Pro B70 32GB cards, per the Red-Weasel/machx-inference-engine GitHub README. It serves an OpenAI compatible endpoint with tool calls and up to 16 parallel requests.

How many agents can it run at once?
It supports agent swarms to 16 with shared prefix caching, per the same README. Respect MiMo and Flash lane caps and design shared prompt heads for best reuse.

Which models are tuned for two B70 cards?
The tuned set lists MiMo V2.6 Flash, DeepSeek V4.1 and V4 Flash, GLM 5.3 Flash, Qwen3.8 Flash, Qwen3.6 35B A3B class, and Qwen3.8 27B with dated README rows, per the same README. Exact dates and rates need a live README check [NEEDS VERIFICATION for exact dates and rates].

What is the v0.2.6 swarm reference?
The repo reported v0.2.6 agent swarm replay is 75 requests in 139.7 seconds, per the README and releases. That is a repo reported scale marker and never our measurement.

What limits should I watch?
Watch shared prefix byte differences, 35B class only scheduling, MiMo and Flash lane caps, tool call edge cases, DeepSeek greedy variance, and first request kernel compile cost, per the same README. Log warm and steady runs separately.

Where is the source code and license?
See the Red-Weasel/machx-inference-engine GitHub README and releases for code, docs, and version notes. The repository profile notes Apache 2.0 licensing with 4 stars and 19 commits as cited in the brief.

Bottom Line

Mach X gives Arc Pro B70 owners a focused local path with C++ and SYCL hand kernels, tensor parallel two card serving, auto lanes, 16 parallel requests, agent swarms to 16 with shared prefix caching, structured tool calls with reasoning_content fields, native Q6_K and Q5_K, and fast safe stops, per the Red-Weasel/machx-inference-engine GitHub README. The seven tuned families with dated two card rows make model choice concrete on the reference hardware. The v0.2.6 replay of 75 requests in 139.7 seconds sets a repo reported scale point for swarm planning.

Limits are part of the value story rather than fine print. Shared prefix byte differences, 35B class only scheduling, MiMo and Flash lane caps, tool call edge cases, DeepSeek greedy variance, and first request kernel compile cost all shape real numbers, per the same README. Small project scope with Apache 2.0 licensing, 4 stars, and 19 commits plus one token default figures means owners should test their own prompts. Pin the tag, respect caps, warm up before timing, and share prefixes across agents.

Start by confirming two B70 32GB cards, installing the tagged engine build, loading one Flash model in Q6_K or Q5_K, serving with auto lanes, and running a fixed 75 request swarm script with shared heads. Compare total seconds across lane counts and prefix shapes with the same tag. Recheck the README and releases before quoting dated rows. That loop turns a promising Arc tuned server into a repeatable local tool.

Sources

All engine facts, kernel notes, server features, tuned model names, two card reference setup, lane behavior, swarm scale, and limits in this article are attributed to the Red-Weasel/machx-inference-engine GitHub README, which describes a C++ and SYCL engine built for Intel Arc, tuned on two Arc Pro B70 32GB cards, with hand XMX, int dot GEMV, and tiled FlashAttention kernels, an OpenAI compatible server with tool calls, tensor parallel multi GPU, auto lanes, 16 parallel requests, agent swarms to 16 with shared prefix caching, structured tool calls and reasoning_content fields, native Q6_K and Q5_K, fast safe stops, seven tuned models with dated benchmark rows, and the listed honest limits. Project profile context of Apache 2.0 licensing with 4 stars and 19 commits comes from the repository profile as cited in the assignment brief. Version replay context of 75 requests in 139.7 seconds for v0.2.6 comes from the same README and releases. See the primary source here: Red-Weasel/machx-inference-engine GitHub README. See version history here: Red-Weasel/machx-inference-engine releases. Any sentence marked [NEEDS VERIFICATION] needs a fresh check against the live README or your own test run before readers treat it as confirmed fact.