Enhancing Language Modelling with RNN and LSTMbased Next Word Prediction
Keywords:
Neural network, Language modelling, Recurrent Neural Network (RNN), Natural Language Processing (NLP), Long Short-Term Memory (LSTM)Abstract
A growing field of study, natural language processing is utilised in many different contexts. We frequently text each other, and we have noticed that whenever we type a message, a recommendation pops up to try to guess what word we will write next. The task of anticipating the next word is known as next word prediction or language modelling. This makes them highly effective for tasks such as language modelling, speech recognition, sentiment analysis, and machine translation. In this study, we will provide a comprehensive overview of RNNs. Additionally, we will explore the limitations and difficulties of RNNs, as well as recent advances such as attention mechanisms that have enhanced their performance. The process for language modelling on a collection of texts now uses artificial neural networks. Recurrent neural networks (RNNs) can use all of the words that have come before to predict the next word, unlike feed-forward networks that can only use a predetermined context length of words.
References
Ganai AF, Khursheed F. Predicting next word using RNN and LSTM cells: Stastical language modeling. In 2019 IEEE 5th International Conference on Image Information Processing (ICIIP). 2019 Nov 15; 469–474.
Kumar R, Sahula V. intelligent Approaches for natural language processing for Indic languages. In 2021 IEEE International Symposium on Smart Electronic Systems (iSES) (Formerly iNiS). 2021 Dec 18; 331–334.
Yin FL, Pan XY, Liu XW, Liu HX. Deep neural network language model research and application overview. In 2015 IEEE 12th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). 2015 Dec 18; 55–60.
Habib MT, Al-Mamun A, Rahman MS, Siddiquee SM, Ahmed F. An exploratory approach to find a novel metric based optimum language model for automatic Bangla word prediction. Int J Intell Syst Appl. 2018 Feb 1; 12(2): 47–54.
Naya C, Kumar A. Next word prediction using machine learning techniques. Adv Eng Sci. 2022; 54(2): 5161–71.
Soam M, Thakur S. Next Word Prediction Using Deep Learning: A Comparative Study. In 2022 IEEE 12th International Conference on Cloud Computing, Data Science & Engineering (Confluence). 2022 Jan 27; 653–658.
Ahirrao M, Joshi Y, Gandhe A, Kotgire S, Deshmukh RG. Phrase Composing Tool using Natural Language Processing. In 2021 IEEE International Conference on Intelligent Technologies (CONIT). 2021 Jun 25; 1–4.
Akandeh A, Salem FM. Slim lstm networks: Lstm_6 and lstm_c6. In 2019 IEEE 62nd International Midwest Symposium on Circuits and Systems (MWSCAS). 2019 Aug 4; 630–633.
Xiao J, Zhou Z. Research progress of RNN language model. In 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA). 2020 Jun 27; 1285–1288.
Barman PP, Boruah A. A RNN based Approach for next word prediction in Assamese Phonetic Transcription. Procedia Comput Sci. 2018 Jan 1; 143: 117–23.
