This is pretty common. It's caused by being unable to connect to certain people. It will persists with that same person until one of you restarts your PS3. Since you are trying to invade, there are probably only a few people in your level range in 1-4 at the time times you are trying this, all of which you are unable to connect to. If this persists for a long time, or if you're trying to be summoned by somebody specific, I'd suggest restarting your PS3.
Also possible, though less common, is that people are invasion-dodging. This is when somebody carries around something like a Red Eye Stone or Shard of the Archstone. Anything that can't be used in multiplayer. When it fades out, they know they're about to be invaded. You can, at this point, load your profile and prevent the invader from connecting to you. They'll get a Failed to Join Session message.
