Where a model runs
Online, on a workstation or on your own servers: what changes for your data, your costs and your uses.
ObjectiveChoose where to run a model according to the sensitivity of the data and the organisation’s constraints.
A model is a file of parameters plus a program that computes with them. It can run in a data centre, on a powerful computer or on an organisation’s servers. Open-weight models, whose parameters are published, make the last two options possible.
Three places, three trade-offs
- Online. The largest models, with no hardware to manage. The question becomes where it is hosted and which law applies.
- On a workstation. The data never leaves the computer. The models have to be smaller or compressed, which reduces their capabilities.
- On your servers. The data stays within your infrastructure, even without Internet access. You need the hardware and a team to operate it.
Compressing a model, called quantisation, stores the parameters with less precision. A quantised model takes much less memory, at the cost of a slight loss of quality that you should measure on your own use cases.
References
- Jiang, Sablayrolles, Roux et al. (2024). Mixtral of Experts. arXiv preprint. arxiv.org/abs/2401.04088