You’re entirely correct, but in theory they can give it a pretty good go, it just requires a lot more computation, developer time, and non-LLM data structures than these companies are willing to spend money on. For any single query, they’d have to get dozens if not hundreds of separate responses from additional LLM instances spun up on the side, many of which would be customized for specific subjects, as well as specialty engines such as Wolfram Alpha for anything directly requiring math.
LLMs in such a system would be used only as modules in a handcrafted algorithm, modules which do exactly what they’re good at in a way that is useful. To give an example, if you pass a specific context to an LLM with the right format of instructions, and then ask it a yes-or-no question, even very small and lightweight models often give the same answer a human would. Like this, human-readable text can be converted into binary switches for an algorithmic state machine with thousands of branches of pre-written logic.
Not only would this probably use an even more insane amount of electricity than the current approach of “build a huge LLM and let it handle everything directly”, it would take much longer to generate responses to novel queries.
I would recommend against pairing Battlemage with a low-spec CPU. As shown by Hardware Canucks, Hardware Unboxed, and others, Intel’s Arc graphics driver overhead is currently much higher than competitors, which means they’re disproportionately affected by having a weaker CPU. This causes the B580 to lose significantly more performance when paired with low-end CPUs than a roughly equivalent Nvidia or AMD card. At the very low end, the difference is especially stark. In some games, the B580 goes from neck-and-neck with a 4060 on a high-end CPU to losing half its performance with a low-end older CPU, while the 4060 only loses about 25%.
If you’re really stuck with a lower-end CPU, it would be far better to get a used midrange AMD or Nvidia GPU from an older product generation for the same price and use that.