nlp/notes.md

What needs to be improved about this repo:

Generalize and standardize the steps in an NLP pipeline into python classes and functions. I can think of these off the top of my head:

• Scraper - get text from the internet to local file
• Cleaner - clean raw text of non-corpus text
• Ngramer - assemble text in python list of lists
• Cfdister - restructure data into a conditional frequency distribution
• Other? - restructure data by other metric (rhyming, similarity, etc.)
• Assembler loop - takes structure above and outputs one word
  • Maybe should wrap in a sentence loop, line-by-line loop, paragraph loop, etc.

Syntax aware generate is actually pretty bad. I think it forces it to be too random. The POS tagging is too error prone and fine-detailed.

Ideas for the future:

Pick one or two lines of the haiku from actual haiku or other poems. Then add a line or two from the corpus (e.g. trump tweets) that both fits the syllables and rhymes with the end(s) of the real poetic line. I think both sources could be ngram generated, but I think it would be ideal if they were picked wholesale from the source. The problem with that approach is that you'd also have to find a common word between the two source extractions so that the sentence doesn't abruptly shift between lines. Or, maybe that's a good thing? I guess I should try both.

Maybe try just switching out the nouns, verbs, adjectives, and adverbs leaving the rest of the sentence structure largely intact after the tree replace?

Use word similarity vectors to construct a sentence (or poem) around a central theme. E.g. construct something like the buffalo sentence: syntactically correct sentences, and maybe even semantically meaningful sentences, but in a totally novel form because of some arbitrary restriction (can only use the word "buffalo", or animal words, or onomatopoeias, or etc.).

Integrate alliteration and rhyming.