What matters?

 What matters?

Six things that matter in LLM in July 2024.

1) Scale of the model, number of parameters: Scale with brute force alone won't work. But the scale does matter depending on the overall goal and the purpose of what the LLM is trying to solve.

 

2) Compute matters: Even more than ever, we need to look at the infrastructures around LLMs. Infrastructure is also one of the main constraints for the near term and strategically provides an advantage to a few Middle East countries.

3) Data, quality & quantity. It remains true that high-quality data with extensive (longer) training is the way. Quantity of the data also matters.

4) Loss function matters: If your loss function isn't sophisticated or incentivizes the "right" thing, you will have limited improvement.

5) Symmetry or architecture: Do you have the correct architecture around your model(s) and data? Inefficient engineering can be costly to the overall performance and output. There are inherent structural weaknesses with each approach. Does your overall design overcome or improve using the right kind of symmetry and engineering?

6) Inductive Bias: Optimization and adding inductive bias for a given level of algorithmic development, computing, data, and architecture matters.