There's many reasons why this is so. For many of the less frequently bought lenses, they are made in smaller batches. A store will get what they have on order and that's it. If you want one, you might have to put in an order, and then wait until the store gets it.
Sometimes, a lens gets really popular and a lot of people start to get it. As it gets harder to find, people talk about how hard it is to get one, and that drives the people who were thinking about it to move more quickly, further driving up the demand.
Sometimes it depends on where you look- the larger stores get the most stock, but they may also have a proportionately larger demand. For instance, many of my local dealers have a shortage on the 35/1.8, but when I wanted one last week, it didn't take me long to find someone who did have it (I had to pay more though).
And of course, sometimes lenses are about to be discontinued. This doesn't seem to happen all that often though.