フリー問題

問題文

In the context of a generative language model, what is the purpose of tokenization?

選択肢

1. To translate text into English so that the model can process it consistently
2. To split text into the units that the model actually processes, so that each unit can be mapped to a numeric identifier
3. To remove personal information from text before it is sent to the model
4. To convert text into a single fixed-length vector that captures its overall meaning