NATURAL LANGUAGE PROCESSING
Natural language processing (NLP) refers to the branch of computer science—and more specifically, the branch of AI or AI—concerned with giving computers the flexibility to grasp text and spoken words in much an equivalent way groups of people can.
What is Natural Learning Processing?
Natural Language Processing is a combination of computational fundamental rule-based modeling of human language with statistical, machine learning, and deep learning models. Together, these technologies enable computers to process human language within the variety of text or voice data and to ‘understand’ its full meaning, complete with the speaker or writer's intent and sentiment.
natural language Processing drives computer programs that translate text from one language to a different, answer spoken commands, and summarize large volumes of text rapidly even in real-time. There’s an honest chance you’ve interacted with NLP within the kind of voice-operated GPS systems, digital assistants,speech-to-text dictation software, customer service Chabot, and other consumer conveniences. Natural Language Processing also plays an important expanding role in enterprise solutions that help streamline business operations, increase employee productivity, and simplify mission-critical business processes.
Natural Language Processing Fields of Use:
Human language is crammed with ambiguities that make it incredibly difficult to write down software that accurately determines the intended meaning of text or voice data. Homonyms, homophones, sarcasm, idioms, metaphors, grammar and usage exceptions, variations in sentence structure—these just a couple of the irregularities of human language that take humans years to find out, but that programmers must teach natural language-driven applications to acknowledge and understand accurately from the beginning if those applications are getting to be useful.
Several NLP tasks break down human text and voice data in ways in which help the pc add up to what it's ingesting. a number of these tasks include the following:
® Speech recognition, also called speech-to-text, is that the task of reliably converting voice data into text data. Speech recognition is required for any application that follows voice commands or answers spoken questions. Things that make speech recognition challenging are that the way people talk—quickly, slurring words together, with varying emphasis and intonation, in several accents, and sometimes using incorrect grammar.
® Part of speech tagging also called grammatical tagging, is that the process of determining a part of speech of a specific word or piece of text that supported its use and context. apart of speech identifies ‘make’ as a verb in ‘I can make a paper plane,’ and as a noun in ‘What make of car does one own?’
® Word sense disambiguation is that the selection of the meaning of a word with multiple meanings through a process of semantic analysis that determines the word that creates the foremost sense within the given context. for instance, word meaning disambiguation helps distinguish the meaning of the verb 'make' in ‘make the grade’ (achieve) vs.‘make a bet’ (place).
® Name identity recognition, or NEM, identifies words or phrases as useful entities. NEM identifies ‘Kentucky’ as a location or ‘Fred’ as a man's name.
® Co-reference solution is that the task of identifying if and when two words ask an equivalent entity. the foremost common example is determining the person or object to which a particular pronoun refers (e.g., ‘she’ = ‘Mary’), but it also can involve identifying a metaphor or an idiom within the text (e.g., an instance during which 'bear' isn't an animal but an outsized hairy person).
® Sentiment analysis attempts to extract subjective qualities—attitudes, emotions, sarcasm, confusion, suspicion—from the text.
® Natural language generation is usually described because the opposite of speech recognition or speech-to-text; it is the task of putting structured information into human language.
® Translation is that the rummage around for the foremost adequate thanks to express a phrase or a word during a different language. the simplest example could also be Google’s translator, a tool that has gradually improved with the passage of your time in the beginning, its performance was very deficient since it used Phrase-Based MT (PBMT). PBMT looks for similar phrases between different languages, which suggests it doesn’t always find phrases with an equivalent meaning; since there are words in one language that don’t exist in another, it’s impossible to translate them correctly. at the present, Google uses Google Neural MT (GNMT), which uses machine learning with NLP to seem for various patterns in languages.
® Chabot’s: are programs that hold conversations with humans. for instance, when someone must buy something from a webshop and has certain questions on the merchandise, the answers they receive are probably generated by a machine.
® Market intelligence: supported what an individual has looked for on the web, it searches for any related ads. an honest example of market intelligence is when an individual looks for a selected product and automatically, ads associated with the merchandise show abreast of social media.
® Question/answer systems: consists of the automated answering of questions by a program. This application of NLP is often easily found in social media chats, calls, or tools like Siri and IBM Watson.
Advantages of Natural Language Processing: -
® Less expensive: employing a program is a smaller amount costly than employing an individual. an individual can take two or 3 times longer than a machine to execute the tasks.
® Usually, call centers have limited staff, which restricts the number of calls that will be answered. By using NLP, better call volume is often handled which suggests client wait times are reduced.
® Easy to implement: within the past, to use NLP, arduous research had to require place regarding the language and lots of tasks had to be implemented manually. In many cases, when it came to translation, it had been necessary to make a kind of dictionary that included words that would be translated into another language. Therefore, it took an extended time to develop. But today, it's easy to seek out pre-trained machine learning models that facilitate different applications of NLP.
Disadvantages of natural language processing:
® Training can take time: if it’s necessary to develop a model with a replacement set of knowledge without employing an are-trained model, it can take weeks to realize an honest performance counting on the quantity of knowledge.
® It’s not 100% reliable every one of the disadvantages of machine learning is that it’s never 100% dependable. There's the likelihood of errors in its prediction and results.
® NLP system doesn’t have an interface that lacks features that allow users to further interact with the system. If it's necessary to develop a model with a replacement one without employing an are-trained model, it can take every week to realize an honest performance depending on the quantity of knowledge.
®The system is made for one and specific task only, it's unable to adapt to new domains and problems due to limited functions
Evolution of natural language processing
While linguistic communication processing isn’t a brand-new science, the technology is rapidly advancing because of an increased interest in human-to-machine communications, plus the availability of massive data, powerful computing, and enhanced algorithms.
As a human, you can speak and write in English, Spanish or Chinese. But a computer’s linguistic communication – referred to as computer code or machine language –is essentially incomprehensible to most people. At your device’s lowest levels, communication occurs not with words but through many zeros and ones that produce logical actions.
Indeed, programmers used punch cards to speak with the primary computers 70 years ago. This manual and arduous process were understood by a comparatively small number of individuals. Now you'll say, “Alexa, I prefer this song,” and a tool playing music in your home will lower the quantity and reply, “OK. Rating saved,” during a humanlike voice. Then it adapts its algorithm to play that song – et al am passionate about it – subsequent time you hear that music station.
Let's take a better check out that interaction. Your device activated when it heard you speak, understood the unspoken intent within the comment, executed action, and provided feedback in an exceedingly well-formed English sentence, beat the space of about five seconds. the entire interaction was made possible by NLP, alongside other AI elements like machine learning and deep learning.
Why is NLP important?
· Large volumes of textual data:
Natural language processing helps computers communicate with humans in their language and scales other language-related tasks. for instance, NLP makes it possible for computers to read text, hear speech, interpret it, measure sentiment, and determine which parts are important.
Today’s machines can analyze more language-based data than humans, without fatigue and in a consistent, unbiased way. Considering the staggering amount of unstructured data that’s generated a day, from medical records to social media, automation is going to be critical to completely analyze text and speech data efficiently.
· Structuring a highly unstructured data source:
Human language is astoundingly complex and diverse. We express ourselves in infinite ways, both verbally and in writing. Not only are there many languages and dialects but within each language may be a unique set of grammar and syntax rules, terms, and slang. once we write, we frequently misspell or abbreviate words, or omit punctuation. once we speak, we've regional accents, and that we mumble, stutter and borrow terms from other languages.
While supervised and unsupervised learning, and specifically deep learning, are now widely used for modeling human language, there’s also a requirement for syntactic and semantic understanding and domain expertise that isn't necessarily present in these machine learning approaches.NLP is vital because it helps resolve ambiguity in language and adds useful numeric structure to the info for several downstream applications, like speech recognition or text analytics.
NLP tools and approaches
Python and the Natural Language Toolkit (NLTK)
The Python programming language provides a good range of tools and libraries for attacking specific NLP tasks. Many of those are found within the tongue Toolkit, or NLTK, an open-source collection of libraries, programs, and education resources for building NLP programs.
The NLTK includes libraries for several of the NLPtasks listed above, plus libraries for subtasks, like sentence parsing, word segmentation, stemming and lemmatization (methods of trimming words right down to their roots), and tokenization (for breaking phrases, sentences, paragraphs, and passages into tokens that help the pc better understand the text). It also includes libraries for implementing capabilities like semantic reasoning, the power to succeed in logical conclusions supported facts extracted from text
Statistical NLP, machine learning, and deep learning
The earliest NLP applications were hand-coded,rules-based systems that would perform certain NLP tasks, but couldn't easily scale to accommodate a seemingly endless stream of exceptions or the increasing volumes of text and voice data.
Enter statistical NLP, which mixes computer algorithms with machine learning and deep learning models to automatically extract, classify, and label elements of text and voice data then assign a statistical likelihood to every possible meaning of these elements. Today, deep learning models and learning techniques supported convolutional neural networks (CNNs)and recurrent neural networks (RNNs) enable NLP systems that 'learn' as they work and extract ever more accurate meaning from huge volumes of raw, unstructured, and unlabeled text and voice data sets.
Natural language processing Cases:
Natural language processing is that the drive behind machine intelligence in many modern real-world applications. Here are a couple of examples:
®Spam detection: you'll not consider spam detection as an NLP solution, but the simplest spam detection technologies use NLP's text classification capabilities to scan emails for language that always indicates spam or phishing. These indicators can include overuse of monetary terms, characteristic bad grammar, threatening language, inappropriate urgency, misspelled company names, and more. Spam detection is one among a couple of NLP problems that experts consider 'mostly solved' (although you'll argue that this doesn't match your email experience).
®Machine translation: Google Translate is an example of widely available NLP technology at work. Truly useful MTinvolves quite replacing words in one language with words of another. Effective translation has got to capture accurately the meaning and tone of the input language and translate it to text with an equivalent meaning and desired impact within the output language. MT tools are making good progress in terms of accuracy. an excellent thanks to testing any MT tool is to translate text to at least one language then back to the first. An oft-cited classic example: shortly ago, translating “The spirit is willing but the flesh is weak” from English to Russian and back yielded “The vodka is sweet but the meat is rotten.” Today, the result's “The spirit desires, but the flesh is weak,” which isn't perfect but inspires far more confidence within the English-to-Russian translation.
®Virtual assistants and chatbots: Virtual assistants like Apple's Siri and Amazon'sAlexa use speech recognition to acknowledge patterns in voice commands and tongue generation to reply with the appropriate action or helpful comments. Chatbots perform equivalent to magic in response to typed text entries. the simplest of those also learn to acknowledge contextual clues about human requests and use them to supply even better responses or options over time. subsequent enhancement for these applications is question answering, the power to reply to our questions—anticipated or not—with relevant and helpful answers in their own words.
®Social media sentiment analysis: NLP has become an important business tool for uncovering hidden data insights from social media channels. Sentiment analysis can analyze language utilized in social media posts, responses, reviews, and more to extract attitudes and emotions in response to products, promotions, and events–information companies can use in product designs, advertising campaigns, and more.
®Text summarization: Text summarization uses NLP techniques to digest huge volumes of digital text and makes summaries and synopses for indexes, research databases, or busy readers who do not have time to read the full text. the simplest text summarization applications use semantic reasoning and tongue generation (NLG) to feature useful context and conclusions to summaries.