I saw this a while back when I saw a post on a dailyprogrammer thread (which was based around language detection) that linked to this lecture by Peter Norvig:
Lecture (Click to View)
The basic idea of this is that you have your text you want to determine the language of, and a variety of text files containing words or sentences of different languages.
If you compare the gzipped original language files with the gzipped language files concatenated with the text you want to verify the language of and record the difference in bytes for each language file, the right language will have the least difference in bytes.
An explaination of why it works is in the lecture above.
For this tutorial, the language files I will be using (and I am only using a few for simplicity) are:
I renamed the files EN.txt, FR.txt and DE.txt respectively, and placed them in a subdirectory named /LanguageFiles
Instead of this being a tutorial outlining what I write line by line and then posting the full python code at the end I have decided to comment the (probably overthought and overcomplicated) source and just post that. I will also include a download to a zip file containing the full project directory including the three language files.
This is how I chose to write my implementation: Pastebin Link
Link to full project directory download and individual file download from my website
References:
Notes:
Lecture (Click to View)
The basic idea of this is that you have your text you want to determine the language of, and a variety of text files containing words or sentences of different languages.
If you compare the gzipped original language files with the gzipped language files concatenated with the text you want to verify the language of and record the difference in bytes for each language file, the right language will have the least difference in bytes.
An explaination of why it works is in the lecture above.
For this tutorial, the language files I will be using (and I am only using a few for simplicity) are:
I renamed the files EN.txt, FR.txt and DE.txt respectively, and placed them in a subdirectory named /LanguageFiles
Instead of this being a tutorial outlining what I write line by line and then posting the full python code at the end I have decided to comment the (probably overthought and overcomplicated) source and just post that. I will also include a download to a zip file containing the full project directory including the three language files.
This is how I chose to write my implementation: Pastebin Link
Link to full project directory download and individual file download from my website
References:
Notes:
- This does not have a 100% success rate (Especially if you start adding more and more languages that are similar to each other)
- Longer text = higher success rate
