https://www.warhistoryonline.com/instant-articles/finding-actors-holocaust.html
They wanted the Reich of the future to be rid of all those humans uncompliant with their vision of an ethnically and racially pure Greater Germany. It led to the execution of more than 300,000 people under the T4 Euthanasia Program, among which was almost the entire European population of people with the growth disorder commonly known as dwarfism.
In a desperate attempt to fill the ranks of short factory workers, the producers tried to employ some overweight children as a substitute, but the idea was abandoned. Although the book contains a few hundred Oompa-Loompa workers, the film had to settle with a mere ten, due to the difficulties in finding actors of suitable stature.