Padma is a Firefox plugin that enables users to read various Indic websites by converting their non-standard text to Unicode text. Padma has supported Anandabazar Patrika (ABP), the largest Bengali newspaper, for more than two years now. However, ABP (also Bartaman Patrika) support in Padma has a major matra-rendering bug which is explained in details here.

Couple of months ago, I made an effort toward fixing the above issue. You can get an improved version of Padma from here. This version resolves most of the matra rendering issue (if not all) for both Anandabazar Patrika (ABP) and Bartaman Patrika.

This Friday I experienced another extreme event in my life. No, it wasn’t any personal event but rather an extreme weather condition. This morning I saw temperature dipping 34 degree Celsius below zero. Yeah, thats right, it was -34 degree Celsius and to make it worse the wind chill factor was -42 degree Celsius. Wind chill factor is roughly a measure of the temperature that you actually feel because of the wind. Incidentally, I am teaching “Math-3503: Differential Equation for Engineers” course at UNB in this semester and I had a class in the morning at 8:30am. So as you can imagine I had the opportunity to feel this extreme temperature head on . I took this screenshot from my Thinkpad before I went out.

Extreme Weather

While waiting at the bus stop, I could literally feel that water vapours are freezing out in my nose as I breath. Later, in my office, I was contrasting this with my days in Chennai. I stayed in Chennai for six years and summer temperature was hitting +44 degree Celsius regularly. In other words, my experience of extreme temperatures now ranges between +44C to -34C. Whoops!!

If you are wondering whether I have the answer to the question above then let me make it clear: No, I don’t. However, if you are among those who often start their work-day by glancing through Google News and who would love to see headlines from Bengali newspapers in it then there is now an alternative. Bengali news headlines (সংবাদ শিরোনাম) from Kashphool.com aims to fill this gap until Google News includes Bengali news sources.

The robot that generates these Bengali news headlines, is a combination of Perl scripts that have been lying around with me for quite sometime. I was thinking of putting them into some good use and Kashphool.com gave me an option. The bot is still in alpha stage but good enough to produce readable headlines. I will release the source codes later. Most of these Bengali news sites uses different dynamic-font scheme and the bot needs to convert them into Unicode before producing these headlines. This headline service gives many Linux users like me, for the first time, an option to read the headlines from almost all Bengali news sources available online.

For last few days I have been experimenting with several Javascript-based virtual keyboards mainly for using in Ankur’s English to Bengali dictionary project. This dictionary project is aiming for a comprehensive English to Bengali dictionary, freely available to everyone. As of now the project ranks highly in Google search for the keywords “English to Bengali dictionary”. This dictionary project relies on user contributions for enhancement of its database. Thus we needed a browser-based solution aimed at helping users in contributing new dictionary entries (in Unicode Bengali) using standard English keyboard and without using any keyboard layout. It also helps to avoid transliterated contribution in English where it should rather be typed in Unicode Bengali.

We have been using bnwebtools for last one year for the purpose. However, this tool has gone in non-GPL direction recently. So we needed a replacement. I was, nevertheless, looking for not just a replacement but also having a next-generation solution .

After playing with few of them I decided to explore the Javascript VirtualKeyboard by Ilya Lebedev. To my surprise, it already had supports for many Indic keyboard layouts. Unfortunately, it didn’t have any Bengali layouts. It appeared that to include a new layout, the layout needs to be described in *.klc file, built using the Microsoft Keyboard Layout Creator tool. Given I had no intension in booting into windows, I wrote a Perl script for creating the *.klc file in the desired format.

To begin with I have converted three Bengali layouts. For Inscript layout, I chose the Baishakhi Inscript used in Baishakhi Linux which is being promoted by Govt of West Bengal (Nevertheless see the recent postings by Sayamindu, Sankarshan, Runa on some controversies surrounding it). Then I converted Ankur’s Probhat layout which I have been using since the beginning. I also converted, another popular Bengali layout Unijoy.

To see this virtual keyboard in action visit Ankur E2B dictionary project or its virtual keyboard demo page. If you have used this virtual keyboard and have any comments/suggestions on it, then please feel free to post them here.

I completed my first post-doc term at IGC, Penn State in the summer. So it was time again for me to move. I really enjoyed Penn State and learned a lot about not just doing science but also different aspects of life in general. Meanwhile, I had accepted an offer to join the Gravity group at Department of Mathematics and Statistics, University of New Brunswick for my second post-doc term. Consequently, me and Nadira had to move from USA to Canada.

In between however, we visited India for a month and half. We were away from home for more than an year. It was the longest ‘away from home’ period for both of us. So we were looking forward to this visit very eagerly.

We had tickets on Lufthansa airlines to Kolkata via Philadelphia and Frankfurt. Our flights were mostly uneventful until we boarded our final flight to Kolkata. Due to a ground staff strike at Frankfurt (yeah, it happened there too )  we were asked to go on ‘fast unto destination’ (sort of)  as there were not enough food on-board. Nevertheless, we were lucky in a sense that our flight at least took off from Frankfurt instead of getting cancelled.  Our joy ride continued even at Kolkata. We were the last passengers of the Lufthansa flight to leave the terminal. We waited till the end to confirm, yes, for the first time, my baggage didn’t arrive with me.  Of course there are more to this story but here I decided not to stray.

For first few days I stayed in my in-laws house at Haldia before I left  for Chennai. The reason was to deliver my PhD viva voce at IMSc. I was about to join for my second post-doc term, but ironically my PhD formality wasn’t complete. After the formal viva and along with a ’sweet-party’ at IMSc, my name officially got two more letter in it. I also gave a technical talk on the quantum gravity stuffs that I have been working on in the recent past.

After another three consecutive but brief professional visits and talks at CTP Jamia, Delhi, SN Bose Center, Kolkata and Saha Institute, Kolkata , finally I was on-board Kanchanjangha express for Malda. Nadira also joined me back at Kolkata. After seven hours of train journey, we got-off at New Farakka junction where my sister and her son were waiting to receive us.

During our stay there, I was on the move for most of the time. There were so many places to visit but so little time I had! All of a sudden, I realized: the time is up. For some unknown reasons, every time I have to leave home, I go through a fight with my inner self. It’s strange kind of emotional dilemma. It has been going on for last sixteen years, ever since I left home to stay in school hostel. Finally, the dreaded moment arrived. The train started to move and I couldn’t pull up my head. Emotions were running high all around and it appeared difficult for me to even glance on the faces of near ones. I felt like crying myself but couldn’t. It reminded me the question again: why does one leave home?

By now, we are slowly settling down in the capital city of New Brunswick province, Fredericton and I am gradually getting back to my full working schedule. Nevertheless, those moments of emotional turbulence often linger in my mind. What lies ahead? Only time will tell.

After a gap of almost two years, I am happy to announce the second official release (version 0.2.0) of Anubadok a free (as in freedom) machine translation system for English to Bengali. Anubadok is written in Perl and it uses Penn Treebank annotation system for natural language processing. To run Anubadok 0.2.0, you need to have Part-of-Speech tagger GPoSTTL installed in your system. The Anubadok system can be accessed online using the interface Anubadok Online run by Ankur.

First official release (ver. 0.1) of Anubadok was an experimental release which mainly served as a proof-of-concept for an open-source English to Bengali machine translation system.

With the release of version 0.2.0, I am glad to upgrade its official tag from “an experimental software” to “a software under development” with clear-and-specific implementation targets. However given the nature of the project, there are no specific time-frames for future releases. Further, given machine translation is considered an open research topic in Computational Linguistic, you should expect to see some surprises even for well implemented situations. Specially, if you are comparing results of machine translations with human translations.

In English, there are four types of sentences: Declarative, Imperative, Interrogative and Exclamatory. These sentence types further fall into four basic sentence type: Simple, Compound, Complex and Compound-Complex.

The table below gives approximate status of implementation for each sentence type in the current release and inversely it gives the targets for future implementations.

W: Well implemented
M: Moderately implemented
N: Not/Not-well implemented

Anubadok does not yet have any code to handle Complex or Compound-Complex sentences, not even moderately. This is where next push for development is needed.

Few other salient features of this release:

• The execution method of Anubadok system has been re-written. Anubadok itself has been implemented as Perl module. This means one can now access Anubadok in a Perl program directly by including Anubadok libraries (Perl modules) or in any other program by using appropriate Perl module wrapper.
• The notion of “testsuites” has been introduced for Anubadok. For a given English sentence, it compares a machine translated sentence with the expected Bengali sentence. This is quite an useful tool while adding new features or doing some experimentations as it would ensure that already implemented algorithm are not affected.
• Anubadok system can now handle several kinds of input documents including plain text files, any XML documents, HTML files with in-line javascript, CSS. Further, as earlier, it is capable of translating Portable Object (PO) files directly.
• Anubadok packaging has been completely reorganized to ensure that it has the basic structure of a standard Perl package. Consequently, Anubadok can be installed following the method of standard Perl module installation.
• Anubadok-0.2.0 comes with an updated dictionary having 15K+ entries in its database. This is almost double the number of entries it had in 0.1 release. Credit for this goes to all the contributors of Ankur English to Bengali dictionary project. Anubadok’s dictionary are now updated regularly using database dumps of Ankur E2B dictionary.
• Anubadok has now moved to its new website hosted by SourceForge.

  http://anubadok.sourceforge.net

  Latest source codes of Anubadok can be downloaded from the “trunk” branch of its SVN repository.

• Anubadok Online, the online interface to Anubadok system, has been upgraded substantially. It runs directly using SVN version of Anubadok engine. User contributed new entries though this interface are submitted automatically to Ankur E2B dictionary project.
• A brief document is now available for download as a PDF file from its website. It describes the internal working and the algorithm used by Anubadok system by considering specific example sentence.

[Update: Please see at the bottom of this post for a link to an improved version of Padma.]

Anandabazar Patrika (ABP) and Bartaman Patrika (BP) are two (among big four) well-known Bengali news papers that are published from West Bengal, India. In the Internet era, their online versions are not just a matter of convenience rather the only route of access for many of us. Unfortunately, their online versions continue to live in the past by using non-standard, ancient dynamic font technology instead of upgrading to standard Unicode.

The worst part is that to view their website you need to have Internet Explorer installed in your machine. So if you are Linux, Mac (or any non-Windows) users then you are left at your own.

Fortunately, there is now a simple way for Firefox users in Linux and Mac to read these websites using a Mozilla extension named Padma by Nagarjuna Venna and his team. To get Padma working, (a) you need to have Unicode Bengali font (Linux users may already have one. Mac users can get one from Ekushey), (b) you need to have Firefox (version 3 is recommended for Linux but must have for Mac), and (c) and you need to install Padma.

Padma can transform given non-standard encoding to standard Unicode on the fly. Of course, for Padma to work, it must know the font-encoding of the particular website.

As it turns out, I wrote support for ABP in Padma more than a year ago. My job was made simple by an earlier CGI program by Tanmoy Bhattacharya who had already decoded font-mapping for ABP. Couple of months ago, I also added support for Bartaman Patrika in Padma. So, courtesy Tanmoy’s font-map decoding, latest version of Padma (0.4.13) supports both ABP and BP.

There is a known issue of incorrect rendering of Bengali Matras in certain situations. See for example Runa-Sankarshan’s photostream here. Many of these were due to a simple bug and has been fixed in the latest version (0.4.13). However, fixing of the remaining requires significant changes in Padma. ABP and BP both use three different fonts simultaneously. Most ligatures often come from 2nd and 3rd font whereas Matras come from the 1st font. Padma transforms each font separately and doesn’t merge these different fonts elements into a single element. This leads to the incorrect rendering which is hard to solve without changing the core of Padma.

The Bigger Issue however is the need for Padma itself. I tend to agree with the concerns expressed by Sankarshan in a discussion thread here. The real question is then how long are these websites going to keep themselves confined using their own non-standard encoding?

This led me to wonder: don’t their technical staffs realize what they are missing by not upgrading to Unicode? Firstly, by upgrading to Unicode they could readily expand their current user base. Secondly, the use of Unicode will make their contents search-able in search engine like Google. This could lead to additional search-engine generated revenue for them. The number of Bengali internet users is going to increase in coming future, and a significant portion of new internet users will be coming from the interior part. Undoubtedly, many of these users will be more comfortable in searching using Bengali keywords. Thirdly, by continuing the use of non-standard encoding, they are piling up their archive with non-standard contents which would require a big effort by them to bring into standard form. So, in my humble opinion, it would be prudent decision for them to upgrade their website to use Unicode sooner than later.

Nevertheless, there is now a positive sign that Star Ananda, a sister group of Anandabazar Patrika, has started using Unicode (though their defined “charset” doesn’t say so) for their Bengali website. I hope, this marks the beginning of change.

…

Update (May 9, 2009): Please see this post for an update on the above mentioned incorrect rendering issue.

If you are familiar with LaTeX and interested in writing Bengali documents (or any other Indic documents) using it then you might find this post relevant.

In brief, LaTeX is a type-setting system which is widely used in scientific communities for preparing technical articles.

Here, instead, I will be interested in writing Bengali documents using LaTeX. The first Bengali typesetting system using LaTeX was the well known Bangtex package by Palash B. Pal. Bangtex does produce good quality documents.

However, main inconvenience in using Bangtex package is its input method. One needs to follow a specific transliteration scheme. This suffices if you want to create PS/PDF file at the end. However if you also want to create html pages for the same documents using latex2html then you are out of luck.

Now wouldn’t it be great if you could type your Bengali (or any other Indic) documents using standard Unicode in LaTeX as you would do with any other application?

Here comes the new revolution, XeTeX, a new type-setting system based on a merger of Donald Knuth’s TeX system with Unicode and modern font (OT/AAT) technologies. This means you no longer need to have special metafonts for your language to be present in the system. Instead you can use standard OTF/TTF fonts those are available in your system, for LaTeX as well.

No more talking… here goes the image of output PDF file.

Above document is created using the following latex source file

%
% This file is a sample tex file to illustrate use of
% XeTeX in typesetting Bengali document.
% License: GNU FDL: (C) Golam Mortuza Hossain, 2008
%
%
% Define fonts that you want to use
%
\font\serif=”FreeSerif:script=beng”
\font\serifbb=”FreeSerif:script=beng” at 24pt
\font\deffont=”FreeSerif:script=beng” at 14pt
\font\mukti=”Mukti Narrow Bold:script=beng”

%
% “script=beng” in above lines ensures that appropriate
% complex text rendering engines are used for proper
% rendering of Bengali.
%

\title{\bf\serifbb পাগলা দাশু}
\author{\bf\serif সুকুমার রায়}
\date{}

\begin{document}
\deffont %Default font used for the document
\maketitle

আমাদের স্কুলের যত ছাত্র তাহার মধ্যে এমন কেহই ছিল না, যে পাগলা দাশুকে
না চিনে। যে লোক আর কাহাকেও জানে না, সেও সকলের আগে পাগলা দাশুকে চিনিয়া
লয়। সেবার একজন নূতন দারোয়ান আসিল, একেবারে আনকোরা পাড়াগেঁয়ে লোক, কিন্তু
প্রথম যখন সে পাগলা দাশুর নাম শুনিল, তখনই সে আন্দাজে ঠিক ধরিয়া লইল যে,
এই ব্যক্তিই পাগলা দাশু। কারণ তার মুখের চেহারায়, কথাবার্তায়, চলনে চালনে
বোঝা যাইত যে তাহার মাথায় একটু ‘ছিট’ আছে। তাহার চোখদুটি গোল-গোল,
কানদুটা অনাবশ্যক রকমের বড়, মাথায় এক বস্তা ঝাঁকড়া চুল। চেহারাটা দেখিলেই
মনে হয়—

\vskip 10mm

% Specific font can be used for any part of the document
{\mukti বিঃদ্রঃ এই বাংলা পাতাটি লেখার জন্য XeTeX ব্যবহার করা হয়েছে।}

\end{document}

sudo apt-get install texlive-xetex

to install it. To test above sample tex file you can cut-n-paste to a file say sample.tex then compile it using

xelatex sample.tex

If everthying goes fine then you should get sample.pdf as your output document. In case you need to edit the latex source file, you can use any editor of your choice like kate, gedit or even vim. Of course, you can also use specialized LaTeX editor like kile (KDE Integrated LaTeX Environment) where you can view output document simply in one click!!