フリー問題

問題文

A team enables mixed-precision training. Which pair of measures keeps the numerics sound?

選択肢

1. Keep every tensor in the reduced format and lower the learning rate to compensate, because the loss of precision behaves like additional noise in the gradient and a smaller step size is the standard way to absorb that noise during a training run.
2. Keep a high-precision master copy of the weights and scale the loss so small gradients do not underflow.
3. Round all activations to the nearest representable value at every layer boundary and keep one copy of the weights.
4. Disable the reduced format for the backward pass only.