Yes - I wouldn't even be able to name my Noble Phantasm if that were the case...

[Which could lead to greater issues for himself and Gilgamesh]

If this is like the Witch's game, the removal of our ability is to make things more even or give our entity an advantage but they would've done it soon as they notice our presence - unless... they were unable to do so until - that man's SOUL was claimed.