IMO, you let them borrow it, like everything you let people borrow if you happen to need it back it is not rude to ask for it back. So i say go ahead and ask for it back, they can always get another one, what is baby sleeping in now since they arent using the crib?
I've had alot of people ask me for some of my baby stuff after each one of my kids. I held onto everything for that reason!!! I don't know what I would do. Hopefully they will be understanding and say its not a big deal to switch it.
I wouldn't say it's rude, but maybe sit down and talk with them about letting them use it in till you are going to need ?? Give them time to try to find one or see how they feel about it. If it wasn't said if it was a gift or for use then I would talk to them.