How do I convert a file's format from Unicode to ASCII using Python?

I think this is a deeper issue than you realize. Simply changing the file from Unicode into ASCII is easy, however, getting all of the Unicode characters to translate into reasonable ASCII counterparts (many letters are not available in both encodings) is another. This Python Unicode tutorial may give you a better idea of what happens to Unicode strings that are translated to ASCII: http://www.reportlab.com/i18n/python_unicode_tutorial.html Here's a useful quote from the site: Python 1.6 also gets a "unicode" built-in function, to which you can specify the encoding: > >>> unicode('hello') u'hello' > >>> unicode('hello', 'ascii') u'hello' > >>> unicode('hello', 'iso-8859-1') u'hello' > >>> All three of these return the same thing, since the characters in 'Hello' are common to all three encodings. Now let's encode something with a European accent, which is outside of ASCII. What you see at a console may depend on your operating system locale; Windows lets me type in ISO-Latin-1. > >>> a = unicode('André','latin-1') > >>> a u'Andr\202' If you can't type an acute letter e, you can enter the string 'Andr\202', which is unambiguous. Unicode supports all the common operations such as iteration and splitting. We won't run over them here. IMHO, PConroy has a good answer. I'd vote him up, but I'm out for today :D

Like this: uc = open(filename).read().decode('utf8') ascii = uc.decode('ascii') Note, however, that this will fail with a UnicodeDecodeError exception if there are any characters that can't be converted to ASCII. EDIT: As Pete Karl just pointed out, there is no one-to-one mapping from Unicode to ASCII. So some characters simply can't be converted in an information-preserving way. Moreover, standard ASCII is more or less a subset of UTF-8, so you don't really even need to do any decoding.

Here's some simple (and stupid) code to do encoding translation. I'm assuming (but you shouldn't) that the input file is in UTF-16 (Windows calls this simply 'Unicode'). input_codec = 'UTF-16' output_codec = 'ASCII' unicode_file = open('filename') unicode_data = unicode_file.read().decode(input_codec) ascii_file = open('new filename', 'w') ascii_file.write(unicode_data.write(unicode_data.encode(output_codec))) Note that this will not work if there are any characters in the Unicode file that are not also ASCII characters. You can do the following to turn unrecognized characters into '?'s: ascii_file.write(unicode_data.write(unicode_data.encode(output_codec, 'replace'))) Check out http://docs.python.org/library/stdtypes.html#str.encode for more simple choices. If you need to do anything more sophisticated, you may wish to check out http://code.activestate.com/recipes/251871/ at the Python Cookbook.

By the way, these is a linux command iconv to do this kind of job. iconv -f utf8 -t ascii <input.txt >output.txt

It's important to note that there is no 'Unicode' file format. Unicode can be encoded to bytes in several different ways. Most commonly UTF-8 or UTF-16. You'll need to know which one your 3rd-party tool is outputting. Once you know that, converting between different encodings is pretty easy: in_file = open("myfile.txt", "rb") out_file = open("mynewfile.txt", "wb") in_byte_string = in_file.read() unicode_string = bytestring.decode('UTF-16') out_byte_string = unicode_string.encode('ASCII') out_file.write(out_byte_string) out_file.close() As noted in the other replies, you're probably going to want to supply an error handler to the encode method. Using 'replace' as the error handler is simple, but will mangle your text if it contains characters that cannot be represented in ASCII.

As other posters have noted, ASCII is a subset of unicode. However if you: have a legacy app you don't control the code for that app you're sure your input falls into the ASCII subset Then the example below shows how to do it: mystring = u'bar' type(mystring) <type 'unicode'> myasciistring = (mystring.encode('ASCII')) type(myasciistring) <type 'str'>

For my problem where I just wanted to skip the Non-ascii characters and just output only ascii output, the below solution worked really well: import unicodedata input = open(filename).read().decode('UTF-16') output = unicodedata.normalize('NFKD', input).encode('ASCII', 'ignore')