ProximityPrompt HoldDuration can be changed on the client. The server does not validate HoldDuration either (if I remember correctly), so you can still listen to Triggered without issues on the server.
@RCW3993 or you could have a attribute for the player (proximity prompt gamepass), and when they are near a proximity prompt or using it, if that attribute is set to true it sets the proximity prompt time to whatever shorter time
Pretty sure this is the way to go. To get all of the proximityPrompts I would suggest either applying a tag to all of them and use CollectionService to filter through them all or just put them in a Folder and use GetChildren().