o 38i! @s2dZddlZddlZddlmZmZmZmZmZddl m Z ej ddzddl Z Wney:ddlm Z YnwGdd d Zed kreZgd Zed eD]\ZZZed edededqSeeZedeedeD]Zed eddeddeddeddd qxdSdS)u[ Term Cleaner Module =================== Stop-word temizliği, filtreleme ve normalizasyon N)ListDictTupleSetOptional)Counterz..)configc@s eZdZdZddZd"dededefdd Zd ed edefd d Zd#dededefddZ d$de e eee fde de efddZ d$de ede de efddZde ede efddZ d%de edede efddZde e eee fde efd d!ZdS)& TermCleanerum Terim temizleme ve filtreleme sınıfı. Stop-word, frekans ve güven skoru filtreleri uygular. cCstj|_tj|_gd|_dS)N))z\s+ )z ^\s+|\s+$)z["\']r )z \([^)]*\)r )z \[[^\]]*\]r )[0-9]+r )r ENGLISH_STOP_WORDS en_stopwordsTURKISH_STOP_WORDS tr_stopwordsclean_patternsselfr;/var/www/html/PEPCVSON/terminology-extractor/src/cleaner.py__init__szTermCleaner.__init__Ftermremove_numbersreturncCsN|sdS|}|jD]\}}|dkr|sq t|||}q d|}|S)u Tek bir terimi temizle. Args: term: Temizlenecek terim remove_numbers: Sayıları kaldır Returns: Temizlenmiş terim r r r )striprresubjoinsplit)rrrcleanedpattern replacementrrr clean_term(s  zTermCleaner.clean_termsourcetargetcs|r|sdSt|tjkrdSt|tjkrdS|}|}t|tjkr)dSt|tjkr2dStfdd|Dr?dStfdd|DrLdS|ddrVdS|ddr`dSdS)u Terim çiftinin geçerli olup olmadığını kontrol et. Args: source: Kaynak terim target: Hedef terim Returns: True if valid Fc3|] }|jvVqdSN)lowerr.0wrrr bz,TermCleaner.is_valid_term..c3r&r')r(rr)rrrr,dr-r r T)lenr MIN_TERM_LENGTHrMAX_TERM_WORDSallreplaceisdigit)rr$r% src_words tgt_wordsrrr is_valid_termCs* zTermCleaner.is_valid_termenlangcs8|dkr|jn|j|}fdd|D}d|S)u Terimden stop-word'leri kaldır. Args: term: İşlenecek terim lang: Dil ('en' veya 'tr') Returns: Stop-word'suz terim r7csg|] }|vr|qSr)r(r) stopwordsrr }sz0TermCleaner.remove_stopwords..r )rrrr)rrr8wordsfilteredrr9rremove_stopwordsos  zTermCleaner.remove_stopwordsNtermsmin_freqc Cs|ptj}t}i}|D]#\}}}||f}||d7<||vs+|||kr/|||<q g} |D]\\}}} | |krN| ||| |||fdq6| S)u Terimleri frekansa göre filtrele ve say. Args: terms: List of (source, target, score) tuples min_freq: Minimum frekans Returns: Filtered and counted terms )r$r% frequency confidence)r MIN_FREQUENCYrr( most_commonappend) rr?r@counterscoressrctgtscorekeyresultfreqrrrfilter_by_frequencys(  zTermCleaner.filter_by_frequencymin_confcsptjfdd|DS)u Terimleri güven skoruna göre filtrele. Args: terms: List of term dicts min_conf: Minimum güven skoru Returns: Filtered terms cs g|] }|ddkr|qS)rCr)get)r*trPrrr;s z4TermCleaner.filter_by_confidence..)r MIN_CONFIDENCE)rr?rPrrSrfilter_by_confidences z TermCleaner.filter_by_confidencecCsTi}|D]}|d}||vr|||<q|d||dkr#|||<qt|S)u Tekrarlı terimleri kaldır, en yüksek frekanslı olanı tut. Args: terms: List of term dicts Returns: Deduplicated terms r$rB)r(listvalues)rr?seenrrLrrr deduplicates    zTermCleaner.deduplicateT lowercasecCsHg}|D]}|}|r|d|d<|d|d<||q|S)u Terim case'ini normalize et. Args: terms: List of term dicts lowercase: Küçük harfe çevir Returns: Normalized terms r$r%)copyr(rF)rr?rZrMrnew_termrrrnormalize_cases  zTermCleaner.normalize_casec Csxg}|D]\}}}||}||}|||r!||||fq||}||} || } | jdddd| S)z Batch halinde terimleri temizle ve filtrele. Args: terms: List of (source, target, score) tuples Returns: Cleaned and filtered terms cSs|dS)NrBr)xrrrsz)TermCleaner.clean_batch..T)rLreverse)r#r6rFrOrUrYsort) rr?r rIrJrK clean_src clean_tgtcounted confidentuniquerrr clean_batchs       zTermCleaner.clean_batch)F)r7r')T)__name__ __module__ __qualname____doc__rstrboolr#r6r>rrfloatintrrOrUrYr]rgrrrrr s4, (    (r __main__))zhydraulic pumpzhidrolik pompagffffff?)zHydraulic PumpzHidrolik Pompagq= ףp?)zthe pumppompag?) safety valve emniyet valfig)\(?)rrrsg?)abirGz?)123456rv)r r rvz Test Terms:z z -> z (score: )z Cleaned Terms (z):r$r%z (freq: rBz, conf: rCz.2f)rkrsystypingrrrrr collectionsrpathinsertr ImportErrorr r rhcleaner test_termsprintrIrJrKrgrMr.rrrrrs:   y