Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.62. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

hello @bee-san can be please let me contribute in ur code?

@chetnaaggarwal-ca Sure! Go right ahead :D

i found this dictionary of words using this https://pypi.org/project/english-words/ but it has only 25487 words which is almost negligible compared to the 479826 in the UNIX default word-list. However all the words in this list are meaningful
here is a copy of that file:
https://pastebin.ubuntu.com/p/HzFSDs29SH/
if this dictionary seems fine to y'all, I'll open a PR
(I was looking for the dictionary after @SkeletalDemise suggest a fix along these lines for issue #412 )

I had tried to create a fix with the logic that @bee-san suggested in the issue description by using this tool https://www.english-corpora.org/coca/ but this had a problem. the string "hm" apparently has a frequency almost 15 times higher than more meaningful words like "curate" and many more.

https://raw.githubusercontent.com/jeremy-rifkin/Wordlist/master/master.txt
300k english words

or you could scrape a whole dictionary
https://dictionary.cambridge.org/dictionary/english/

@Itamai the 300k dictionary looks good to me as well

I also found the dictionary that nltk library uses and it has 236736 words and as far as i am able to see they seem to be all meaningful

which one shall i go with ?

@Itamai the 300k dictionary looks good to me as well

I also found the dictionary that nltk library uses and it has 236736 words and as far as i am able to see they seem to be all meaningful

which one shall i go with ?

the github user @chetnaaggarwal-ca got assigned to this issue so we should let her do it. I was just giving some ideas :D
Another idea would be to merge the 300k one and the 235k one you found together and then delete duplicates that should be good enough or if she knows how to write a scraper then just pull all words from an online dictionary

sure thing ! I'll wait for that to finish so i can begin #412 :)

I tried to merge dictionaries previously and it didn't work out so well :( I also tried scraping the speeches from the Houses of Parliament :(

The dictionary is loaded as JSON so {word: 1}. What do you think about scraping words from multiple sources and building a frequency distribution, like so:

{and: 0.21}

So the word "and" appears a lot. And then we add together the frequencies and if they go over a threshold, we use that?

We do a similar thing for stop words / top 1000 words, except this is for every word in the dictionary :)

Oh right
But the issue there would be that we might miss out on several words that may not be in use in today's colloquial lingo but might still be a meaningful word and someone may come across an encrypted version of it.
This also depends on the source that we pick, but whatever source we pick it still has the possibility to miss out on some word right ?

Oh right
But the issue there would be that we might miss out on several words that may not be in use in today's colloquial lingo but might still be a meaningful word and someone may come across an encrypted version of it.
This also depends on the source that we pick, but whatever source we pick it still has the possibility to miss out on some word right ?

If we used English Wikipedia I think it'd cover a significant amount of words, and the words that do not appear we could always do something like this:

if value == 0 but words are in dict:
    return plaintext
elif value > THRESHOLD:
        return plaintext

It's unlikely for any given text to be compromised entirely of only rare words that have never appeared in English Wikipedia before (or rarely appear) :)

awesome !
we could probably also make a manual procedure to check if by any chance that a word has been falsely claimed to be "not common" this isn't likely but is still possible.
what do you think ?

I found this https://github.com/IlyaSemenov/wikipedia-word-frequency where they conveniently calculated the frequency of english words on wikipedia for us (https://github.com/IlyaSemenov/wikipedia-word-frequency/blob/master/results/enwiki-20190320-words-frequency.txt) so perhaps i could use this along with the dictionaries we currently use to figure out "common words" and split them into 2 files of common words and uncommon words which we can then manually filter to see if we need any words from the uncommon file to be in the common file

I have also found this highly reliable source called datamuse who provide an awesome API which uses google's n-gram to provide frequency of words used in several texts, books and several other material that google has access to. https://www.datamuse.com/api/.

for example : https://api.datamuse.com/words?sp=awesome&md=f&max=1

However, they have a cap of 100k requests per day.

We can get over this cap if 4-5 of us run a script to get the frequency of every single word in the dictionary that we are currently using and then merge the data collected into one file. We can then manipulate this data using whatever threshold we require.

This solution (in my opinion) sounds like the most viable one so far. If y'all agree i can get the script done (including the splitting of the file so 4 of us can run it)

hello @bee-san can be please let me contribute in ur code?

Are you still working on this?

Reassigned issue due to no communication :)