How to handle being a job reference for someone not sure I'd recommend?

Using you as a reference without your permission is pretty dodgy. When/if $PRESTIGIOUS_ORGANIZATION calls, decline to give a reference.

I can't help but put myself in this other persons shoes. They probably really want this job, they had nobody else to ask or had already asked them and they said no, they knew it was the wrong thing to do to just put your name but they did it out of desperation; or something along those lines. I would wait, you might not hear anything. If they do call then just be honest, say things that are positive about the person but state that you worked together a while back and not altogether closely but as far as you remember they were a team player, friendly, approachable etc. After the call (or no call in a few weeks) email/text/call the person back and say that you did or did not give the reference but you're not sure if it worked (or would work) in their favour because it was such a long time ago and you don't have many recent examples to give. Maybe it would be best to find someone more recent. I guess as a recent job hunter I know if my dream job came up and I was grasping at straws to find a referee I would be eternally grateful for even a vague reference. You might find yourself in this situation, you never know.

Do you think the careless way in which this person notified you he used you as a reference without permission indicates that it is likely that the "basic issues" are no longer present?

The most upstanding thing to do is to drop a note to the person and say "X org contacted me for a reference, I wish you the best but I only give those for people who've worked for me directly and recently, so you should provide them with a different one." You don't owe them this, but it would be kinder than the alternative, which is to just decline to give the reference if you are called, citing this reason. I do think it would be odd to start poking around about them or asking them for dossiers.

If you weren't the person's boss, the company seeking a reference isn't going to expect you to go in-depth about the person's work accomplishments, career highlights, personal attributes, etc. They just want to make sure the guy has a few friends & former co-workers who are willing to vouch for him as an ok person so they can be reasonably sure he doesn't have some big glaring flaw that has put people so far off that he can't find anyone who will say some positive stuff about him on the phone for 5 minutes. If the person wasn't awful to work with, there is nothing wrong with saying a few generic nice things about him to a prospective employer. What with this lousy job climate, do you really want to be responsible for costing someone their shot at a good job because you remember they used to be late a lot when you worked with them years ago? If they've already put your name down as a reference, requesting that they withdraw it is almost certainly going to cost them the job. The employer is going to see it as a huge red flag, and this doesn't sound like the kind of situation in which a red flag is warranted. Declining to give a reference will have the same effect. If they weren't the kind of awful, pain-in-the-ass or worse co-worker you'd feel guilty letting some other company hire, it would be a decent thing to do to say something like Youremyworld suggested above. Then let the person know afterwards that you are not comfortable providing a reference for them in the future and to please not use you any more.

Why not do the same thing as you would have originally done? Send them a note that suggests they ask someone else, you don't feel comfortable acting as a recommendation. It'll be best for both you and them.

I've been that person who presumptively put someone down as a reference thinking it was fine, only to find out (through backchannels, they never said anything to me) that my former colleague/supervisor was decidedly NOT helping my cause. I wish they had said something to me, because I feel badly that I put them in that position. Drop your former colleague a note that said something along the lines of "I'm sorry, I'm really not comfortable speaking on your behalf to prospective employers. Please find someone else from our former company to list on your applications." And if the organization contacted me, I simply wouldn't take the call, or wouldn't respond to the email (because I'm avoidant like that.)

do you want to be the jerk who torpedoes someone's job opportunity? Do you want to be the jerk who advantages this guy over someone else who deserves the job more and needs it just as much?

This person has put you in a difficult position against your will. If this organization calls, I would just tell them the truth, that you don't really have any useful info to give them; I don't see any need to complicate things any more than that. If you feel like you want to help out the applicant, I would do what quodlibet suggested above; I wouldn't feel obligated, though.