The sample start and loop start at different places can be confusing and may cause this behavior.
Also you may try zooming in to the beginning to find any silence that may get exaggerated.
It may be just a really low pitched sample that has to “get going” first, in that case the higher pitches will have less delay.
Sorry I don’t have any definitive answers.