Help
Options
Didn't know it?
click below
click below
Knew it?
click below
click below
Don't Know
Remaining cards (0)
Know
retry
shuffle
restart
0:00
Embed Code - If you would like this activity on your web page, copy the script below and paste it into your web page.
Normal Size Small Size show me how
Normal Size Small Size show me how
CECS 429
| Question | Answer |
|---|---|
| What is lemma? | Words with the same stem, part of speech, and rough word sense. |
| Lemma vs. wordform — what is the difference? | Same lemma = same underlying word; different wordforms = different inflected forms. |
| What is a type vs. a token? | Type: unique vocabulary item. Token: each occurrence of that item in text. |
| What do N and V represent? | N = total tokens. V = vocabulary/set of unique types. |
| What is Heaps’ Law? | Vocabulary size grows as the number of tokens increases: V = kNᵇ, usually b ≈ 0.67–0.75. |
| What factors can describe where a text comes from? | Writer, time, language, language variety, and function/purpose. |
| What are language variety and code switching? | a particular form of a language. Code switching: switching between languages/varieties. |
| What is genre and what are author demographics? | Genre: type/category of text. Demographics: age, gender, ethnicity, SES, etc. |
| What questions should we ask about a corpus's motivation? | Why was it collected? By whom? Who funded it? |
| What should we know about a corpus's collection process? | How it was sampled, whether there was consent, preprocessing, annotation, language variety, etc. |
| Why does corpus context matter? | It helps us understand what the text represents and analyze the data correctly. |
| What are regular expressions (regex)? | A formal language for specifying patterns in text strings. |
| What do [ ] and | do in regex? | [ ] matches one character from a set; | means OR/disjunction. Example: [wW], cat|dog. |
| What do [A-Z] and [^A-Z] mean? | [A-Z] = uppercase A–Z. [^A-Z] = anything that is NOT uppercase A–Z. |
| When does ^ mean NOT inside [ ]? | Only when ^ is the first character inside the brackets. |
| What are the regex shortcuts \d, \w, and \s? | \d = digit; \w = alphanumeric/underscore; \s = whitespace. |
| What are the regex shortcuts \D, \W, and \S? | \D = non-digit; \W = not alphanumeric/underscore; \S = non-whitespace. |
| What do ., ?, *, and + mean in regex? | . = any character; ? = 0–1; * = 0+; + = 1+. |
| What do ^, $, and \. mean in regex? | ^ = beginning of line; $ = end of line; \. = literal period. |
| What are Type I and Type II errors? | Type I / false positive: matched something incorrectly. Type II / false negative: missed something that should match. |
| Precision vs. recall — what does each reduce? | Precision: minimizes false positives. Recall: minimizes false negatives. |
| Why are regex useful in NLP? | Used for text processing/preprocessing and can also serve as features in ML classifiers. |
| What does s/ do, and what are ( ) used for? | s/ performs substitutions. ( ) groups and captures text for later reference. |
| What do \1 and (?: ) mean? | \1 refers to the first captured group. (?: ) creates a non-capturing group. |
| What are the 3 main steps of text normalization? | Tokenizing words, normalizing word formats, and segmenting sentences. |
| What is tokenization? | Breaking text into individual tokens, such as words or subwords. |
| How can you simply tokenize languages that use spaces? | Split the text wherever spaces occur. |
| Why can’t we always remove punctuation during tokenization? | Punctuation can be part of prices, dates, URLs, hashtags, and emails. |
| What are a clitic and an MWE? | A clitic cannot stand alone; an MWE is multiple words that may act as one unit. |
| Why is tokenization difficult in languages like Chinese? | Chinese does not use spaces to clearly mark word boundaries. |
| What is a hanzi? | A Chinese character that represents a meaning unit called a morpheme. |
| What is subword tokenization? | Breaking words into smaller parts, including subwords and complete words. |
| What are the 3 common subword tokenization algorithms? | BPE, Unigram language modeling, and WordPiece. |
| What are the 2 parts of a subword algorithm? | A token learner and a token segmenter. |
| What does the BPE token learner start with? | A vocabulary containing individual characters. |
| How does BPE decide which tokens to merge? | It repeatedly merges the most frequent adjacent pair in the training corpus. |
| What is the purpose of the end-of-word symbol in BPE? | It helps preserve word boundaries during subword learning. |
| How does BPE segment test data? | It greedily applies learned merges in the order they were learned. |
| Does BPE use test-set frequencies when segmenting? | No. It uses the merges learned from the training data. |
| What is a morpheme? | The smallest meaning-bearing unit of a language. |
| What are stems and affixes? | A stem carries the core meaning; affixes attach to stems and often show grammar. |
| What does a morphological parser do? | It breaks words into morphemes, such as cats → cat + s. |
| What is word normalization? | Putting words or tokens into a standard format. |
| What is case folding? | Converting letters to a standard case, usually lowercase. |
| When can case information be useful in NLP? | Case can provide useful information for sentiment, translation, and information extraction. |
| What is lemmatization? | Converting words to their shared dictionary headword, or lemma. |
| What is stemming? | Crudely reducing words to their stems by removing affixes. |
| What is the Porter Stemmer? | A stemmer that uses a series of rewrite rules to reduce words to stems. |
| Why is sentence segmentation difficult? | A period can mark a sentence, abbreviation, or number. |
| How can a system identify sentence boundaries? | Tokenize first, then classify periods as word-internal or sentence boundaries. |
| What is a probabilistic language model? | A model that assigns probabilities to sentences or sequences of words. |
| What is the goal of a language model? | To compute the probability of a sentence or word sequence. |
| What is the Markov assumption in an n-gram model? | Predict a word using only a limited number of previous words instead of the entire history. |
| What is the difference between unigram, bigram, and n-gram models? | Unigram uses 0 previous words; bigram uses 1; larger n-grams use more previous words. |
Created by:
MinYoongi67