Parsing HTML pages using Mojo::DOM

After my previous post, I successfully wrote a Perl script to parse an XML file that I had retrieved (from an RSS feed) using Perl and cURL. XML parsing and manipulation is simple enough, if you use the XML::Simple module. But what if the web page you are retrieving with Perl and cURL is an HTML page? And what if you have to make some changes to the HTML code before you render it in your browser? This would require a module that allows you to access individual DOM elements and even modify them. That is how I learned about the Mojo::DOM module, which is part of a package called Mojolicious that you can get on CPAN. Mojolicious calls itself "a next generation web framework for the Perl programming language". You can learn more on their website. Installing it on your machine is simple enough, using the command:

ppm install Mojolicious

Meanwhile, I decided to retrieve a particular web site's home page, make a couple of changes in the code, and render the same page in my browser. The changes were: 1) I would remove their Google analytics code, which was enclosed in a <script> tag pair, and (2) I would ensure that all references to stylesheets and images are prefixed with the URL of that particular web site, so that the web page in my browser appears identical to the original. Here are some of the interesting things that I learned about Mojo::DOM while completing this script, explained usingcode snippets:

$dom = Mojo::DOM->new->xml(0)->parse($response_body);

* Here I have created a new Mojo::DOM object, by parsing the response from my cURL call. The xml(0) is to tell Mojo::DOM to use HTML mode instead of XML mode

my @links = $domref->find('[href]')->each;

* Here I am telling Mojo::DOM to return all HTML tags having a 'href' attribute. I can then go through the @links array using foreach()

$_->attr(href => $linkhref);

* Here I am replacing the value of the 'href' attribute of a particular tag to a string $linkhref, which is actually a modified version of the original value

$_->replace_content($styletext);

* Here I am replacing the entire contents of a particular <style> tag pair with a modified version of the original contents. Interestingly the replace_content function has a bug - it replaces all instances of double quotes with the HTML entity code equivalent '&quot;', which may cause problems in the HTML output. So you need to change the HTML entity code back to double quotes before the end of the script

say "$domref";

* Here I am outputting the modified HTML code to the browser. For those who are not aware, say is the Perl equivalent of Java's println. It saves you the trouble of appending a "\n" at the end of every print statement. Using say requires the line "use feature qw(say);" along with your module declarations

Perl & Curl have my Head in a Whirl

I had recently written a PHP script to read a URL using cURL. Today, I thought I'd convert it into a Perl script. PHP has built in support for cURL, using functions like curl_init(), curl_exec() and curl_close(). In Perl, you have to install and use a CPAN module called WWW::Curl. At first glance, it seemed a simple task. But the problem is that on Windows, you can't simply download the module using the usual command "cpan WWW::Curl". You have to download the source file package and manually install the module. For this, you also need the source package of the cURL libraries. You can't just download and use the "curl" binary, the way you would if you just wanted to run the "curl" command from the DOS prompt. Anyway, here are the steps to follow...

* Ensure a C compiler is installed and available on the PATH. If not, get GCC by installing the Perl MinGW package:

ppm install MinGW

* Download the latest cURL libraries for mingw from the cURL website:

http://curl.haxx.se/download.html

* Unzip the folder somewhere (say C:\curl-7.32.0)

* Download the CPAN module WWW::CURL from CPAN:

http://search.cpan.org/~szbalint/WWW-Curl-4.15/

* Unizp the folder somewhere (say on your Desktop). Edit the Makefile.PL, making the following changes:

1. Comment the lines following the comment "This is a hack."

2. Change the "my @includes..." line to:

my @includes = qw(C://curl-7.32.0//include);

3. Change the "my ($cflags..." line to:

my ($cflags,$lflags, $ldflags) = ('-I"C://curl-7.32.0//include"','-L"C://curl-7.32.0//lib"','-lcurldll -lcurl');

 4. Change the "open(H_IN, "-|", "cpp"" line to:

my $cmd = "cpp $curl_h";
open(H_IN, $cmd) and $has_cpp++;

5. Change the "'LIBS'         => "$ldflags $lflags..." line to:

'LIBS'         => "$lflags $ldflags",        # e.g., '-lm'

* Note that this last change is very important, or else you'll spend 2 hours debugging a linker error - as I unfortunately did

* Run the command:

perl Makefile.PL

* Run the command:

dmake

* Run the command:

dmake install

* Try running your test Perl script. If you get an error, usually as a popup, that says "The procedure entry point ERR_remove_thread_state...", just add "C:\curl-7.32.0\bin" at the beginning of your PATH. This error occurs when your system picks up a wrong version of "libeay32.dll", probably from your installation of SSL VPN or some other application

* Another issue you could face is that your Perl script works fine from the command line but when you run it in Apache, you get an internal server error while loading the WWW::Curl Perl Module, which you can confirm by checking the Apache error log. This is because the "C:\curl-7.32.0\lib" folder is not in your PATH, and needs to be added

Now you're ready to go - write that Perl program to read a URL using cURL. Thanks to Niranjan Prithivraj, whose post was partially helpful during my installation. Hope this post would be useful to someone else, just like Niranjan's post helped me

Defeat Browser Hijackers!

Ever faced a situation where you were careless while installing some software (mostly a free utility) and your browser got hijacked by ask.com? It not only changes your home page and default search provider, it also prevents you from changing the home page back to what you want. So, even if you change the home page from the "Settings" menu, you'll find that "search.ask.com" opens up the next time your start the browser. After an hour of effort, I finally found a permanent fix... Go to "Add/Remove Programs" and uninstall the application called "Movies Toolbar for Chrome (Dist. by Bandoo Media, Inc.)". Then go to your Chrome settings and change the home page once again, this time for good

A Small XSLTip

While making some minor changes to my website recently, I encountered an interesting problem - that of displaying a URL on an XML-XSLT web page. But before I get into the problem and the solution, time for some introductions for the benefit of newbies...

XML: Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. It is a textual data format with strong Unicode support (more on Wikipedia)

XSLT: Extensible Stylesheet Language Transformations (XSLT ) is a language for transforming XML documents into other XML documents, or other objects such as HTML, plain text or XSL Formatting Objects for processing into other formats (more on Wikipedia)

My XML code was as follows:

 <lines>
  <line name="Company" linetext="Capgemini Consulting India Pvt. Ltd." />
  <line name="Description" linetext="Capgemini is a global leader in consulting, technology, outsourcing and local professional services. It employs more than 83,000 employees globally." />
  <line name="My tenure" linetext="2004-2012" />
  <line name="My role" linetext="Manager" />
  <line name="Comments" linetext="Cap was my first big break and the career growth I got was phenomenal. Indeed, the Capgemini stint totally changed me at a personal level." />
  <line name="Website" linetext="http://in.capgemini.com" />
 </lines>

The challenge was to ensure that the website name appeared as a valid URL, whereas the other lines appeared as plain text. This meant that the highlighted "line" with the "name" of "Website" had to be processed in a different way from all the other "lines". After some web searching, I came up with the following XSLT code for processing the "line" tags:

 <xsl:template match="line">
  <tr>
   <td width="30%" align="right" valign="middle" class="itemtext">
    <b><xsl:value-of select="@name" />:</b>
   </td>
   <td width="70%" align="left" valign="middle" class="itemtext">
    <xsl:if test="@name='Website'">
     <a href="{@linetext}"> 
      <xsl:value-of select="@linetext"/>
     </a>
    </xsl:if>
    <xsl:if test="@name!='Website'">
     <xsl:value-of select="@linetext" />
    </xsl:if>
   </td>
  </tr>
  <tr>
   <td colspan="2" align="center" valign="middle">&#160;</td>
  </tr>
 </xsl:template>

Hopefully the above XSLT code is self-explanatory. If not, feel free to contact me for a discussion

Tips on Converting PHP to Perl-CGI

The story today is very similar to yesterday's. Last month I created a new butterfly identification page for my personal website, using PHP and SQLite. By choosing some colors and patterns, newbies (in the butterfly field) can identify any butterflies they may have seen in the field. The page worked well on my desktop but when I uploaded it to Tripod, I realized that they support only Perl and not PHP. So I decided to convert the PHP page to Perl, without having to recode it from scratch. I started by making a copy of the ".php" file and renaming it to ".cgi". Then I made the below changes to convert the PHP code to Perl. Hope these tips would be useful to someone who reads this post.

* Added the Perl shebang line at the top of the page
* Added the standard line to print the content type
* Enclosed all other HTML code within multi-line prints
* Replaced all instances of the PHP "echo" with the Perl "print"
* Replaced the database connection PHP code with the corresponding Perl code
* Replaced the database query PHP code with the corresponding Perl code
* Added a subroutine for getting the CGI parameters, something that PHP automatically did
* Replaced the PHP built-in hash "%POST" to the hash "%form" created by my subroutine
* Replaced the square brackets used for PHP hashes with the curly brackets used in Perl
* Removed the single or double quotes that were used to reference the hash values in PHP
* Replaced the not equal to operator "<>" of PHP with the Perl equivalent "ne"

P.S.: After all that hard work, I found that Tripod does not support the use of DBI.pm. So finally I had to replace the dynamic butterfly identification page with an XML-XSLT page where the readers have to look at a table giving colors and patterns to try and figure out which butterfly they had seen. Well, such is life!

Tips on Converting Python-CGI to Perl-CGI

Last month I created a new feedback page for my personal website, using Python. It worked well on my desktop but when I uploaded it to Tripod, I realized that they support only Perl and not Python. So I decided to convert the feedback page to Perl, without having to recode it from scratch. The groundwork for this took some time, as I had to install Active Perl and brush up my Perl knowledge by tweaking a few old Perl-CGI scripts. Today I created a copy of the CGI script and made the below changes to convert it from Python to Perl. Hope these tips would be useful to someone who reads this post.

* Converted the shebang line from #!/Python27/python to #!/Perl/bin/perl
* Added a semi-colon after every line of Python (now Perl) code
* Added a $ sign before every Python (now Perl) scalar variable
* Converted the multi-line prints from Python format (print """) to Perl format (print <<HTML)
* Removed the Python import line for "sys"
* Removed the Python import line for "time". Used Perl's localtime() function to get date and time
* Removed the Python import line for "cgi". Used a downloaded Perl code snippet to get the CGI parameters, choosing  one that takes care of both GET and POST parameters
* Converted the Python "if-elif-else" syntax to the equivalent Perl "if-elsif-else" syntax
* Converted the Python "try-except" code for opening a text file to the normal Perl file opening code, as Perl does not currently support "try-except"
* Converted the single-line file open and read code of Python to the equivalent Perl code
* Converted the Python syntax [filename.write("abc")] for writing to text file to the equivalent Perl syntax [print FILENAME "abc"]
* Replaced Python's text concatenation operator "+" with the equivalent Perl operator "."
* Removed Python's str() function, as Perl automatically takes care of converting variables based on the context
* Ensured use of "eq" operator instead of "==" operator in "if" conditions involving string variables

Introduction to SharePoint

A part of our Projects team has been spun off into a SharePoint team, which will adapt and implement a particular CRM template for Indian as well as overseas customers. Since our architect is also a part of this new team, he was able to provide some Web links to help me understand what SharePoint is all about

Microsoft SharePoint is a Web application platform. Simply put, it is a tool to store, to manage and to retrieve information. This information could be text or MS Office documents or other files, like videos. The main strength of SharePoint is that even non-technical users can use it as well as develop applications using it

Initially, SharePoint was used for content management and document management on Intranets but recent versions are much more powerful. You can use Sites to store and manage information, and share it on the Intranet or Internet. You can use Communities to collaborate with colleagues. You can use Content management, including integration with your existing Microsoft Office documents. Search allows you to retrieve information quickly and to find people within your organization. Insights allow you to effectively obtain information stored in a different repository, within a certain useful context. SharePoint enables no-code integration of data, documents and processes to create Composite applications

The most basic element in SharePoint is an Item (calendar entry, task, contact) or a Document (Word, Excel, PDF). Both Items and Documents can have metadata associated with them. They can be arranged using Folders

Items are stored in Lists while Documents are stored in Libraries. A List or Library uses Views to display information. VIews depend on context but the standard view is as rows and columns. Each row represents an Item or Document while its metadata is displayed in the columns. Metadata can be grouped into Content Types

Each List or Library is part of a Site, which is a container for data. Sites can be Team Sites, Project Sites or Internet Web Sites. They can be arranged in hierarchies and collected into Site Collections with a Root Site at the top and common site settings. Each Site has a default page (Homepage), along with various other Pages

Sites are viewed using a web browser. Hence there are Web Applications, which can contain several Sites or Site Collections and are hosted on Web Servers. Multiple SharePoint servers can be grouped into a Sharepoint Farm

In short, SharePoint provides a solution to all your information management needs with a secure and remote access. Andreas Glaser's website provides some good SharePoint tutorials that are very useful for beginners. Note that the content of this post has been adapted from his pages, and the SharePoint Wikipedia page