Yes, this is actually an important step. You need to select the multi-reference field as the collection when adding it to your template page.
You should only see the sizes for that particular item.
This collection requires another link to the current item so don’t forget it!