https://en.wikipedia.org/wiki/Chagossians says "The Chagossians are a mix of African, Indian and Malay descent.[4] The French brought some to the Chagos Islands as slaves from Mauritius in 1786. Others arrived as fishermen, farmers, and coconut plantation workers during the 19th century."
The slaves brought there by the French predate British control.
https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians adds that before 1835, the "workers brought to the island by the British" were not really "workers" but forcibly transported slaves, in the same sense that the slaves the English brought to Jamaica were "workers."