mathstodon.xyz is one of the many independent Mastodon servers you can use to participate in the fediverse.
A Mastodon instance for maths people. We have LaTeX rendering in the web interface!

Server stats:

2.8K
active users

#benchmark

4 posts4 participants1 post today

We benchmarked 2k+ cloud servers for LLM inference speed (prompt processing and text generation) using models ranging from 135M to 70B parameters! 🤖

Read tech details, use our open-source tools, learn from the results at sparecores.com/article/llm-inf

sparecores.comLLM Inference Speed BenchmarksWe measured the prompt processing and text generation speed of different LLMs on 2000+ cloud servers.

Did benchmark my two laptops to see how they compare in terms of performance. All tests done under Fedora 42 with the Kernel 6.14 with power-profiles-daemon set to performance.

- ThinkPad T14s Gen4 AMD (Ryzen 7 PRO 7840U)

- ThinkPad X1 Carbon Gen12 (Core Ultra 7 165U)

In CPU power, the results were rather similar with the AMD Ryzen just slightly faster in single- and multi-core performance.

In GPU performance, the AMD Radeon 780M did anihilate the Intel graphics. Three(!) times the score and way faster in every test.

T14s-AMD:
CPU: browser.geekbench.com/v6/cpu/1
GPU: browser.geekbench.com/v6/compu

X1 Carbon-Intel:
CPU: browser.geekbench.com/v6/cpu/1
GPU: browser.geekbench.com/v6/compu

browser.geekbench.comLENOVO 21F8CTO1WW - GeekbenchBenchmark results for a LENOVO 21F8CTO1WW with an AMD Ryzen 7 PRO 7840U processor.

#Mozilla stellt #Benchmark für lokale #KI vor.

Mozilla stellt mit #LocalScore einen einfach einsetzbaren #Benchmark für lokale #KI vor. Der Benchmark ist im Rahmen des Projekts Mozilla Builders entstanden, über das Mozilla-Blogger Sören Hentzschel im vergangenen Jahr berichtet hat. Im Rahmen der Initiative Mozilla Builders Accelerator wurden Förderungen für Open-Source-Projekte zum Thema Lokale KI ausgelobt.

linuxnews.de/mozilla-stellt-be

Llamafile
LinuxNews.de · Mozilla stellt Benchmark für lokale KI vor
More from LinuxNews.de
Continued thread

Indeed, the CPU-only performance is even worse. The LocalScore on the tiny 1B model is only 16, with a text generation speed of 7.7 tokens/second.

localscore.ai/result/235

Let's see if I can run this on a Raspberry Pi for comparison...

www.localscore.aiLocalScore - Test #235 ResultsLocalScore benchmark results for test #235. This is for the accelerator Intel Core i7-8550U CPU @ 1.80GHz (skylake)

My hobby: running LocalScore.ai to benchmark how fast (ehm) my 2018 laptop runs a tiny 1B LLM. The laptop has a NVIDIA MX150 mobile GPU, 2GB VRAM. I guess it was intended for Photoshop filters or CAD stuff.

I got a LocalScore of 101 on the tiny model using the GPU (13.5 tokens/second for generation). A value of around 1000 is considered passable.

localscore.ai/accelerator/234

Still, I think it's marginally better than CPU-only on the same laptop.

www.localscore.aiNVIDIA GeForce MX150 ResultsLocalScore benchmark results for NVIDIA GeForce MX150 with 2GB of memory.