Pashto script and graphics detection in camera captured Pashto document images using deep learning model

Layout analysis and classification

O’Gorman & Kasturi (1995) introduced the document spectrum, for structural page layout analysis. The introduced approach is based on the bottom-up, nearest-neighbor clustering of page sections. The developed approach results in an absolute determination of skew, between-line spacing, and within-line, and finds text blocks and text lines. This method is more beneficial across several other methods in three basic aspects, (1) fairness from different text spacing, (2) freedom of a skew edge (3) the processing capability on confined areas of several text adjustments in the corresponding image.

Simon, Pret & Johnson (1997) developed a new method for document layout analysis based on bottom-up approach. Their algorithm’s implementation was done in the CLiDE (Chemical Literature Data Extraction) system (http://chem.leeds.ac.uk/ICAMS/CliDE.html). Their method is based on Kruskal’s algorithm. To create the physical page formation they used a special distance-metric among the segments. The achieved efficacy of the method by considering the $98$ test images with an error rate of $1$%.

Breuel (2003) presented some novel algorithms and statistical methods for layout analysis. They have assessed their method/approach on a subset of the UW3 (University of Washington Database3) database. Their algorithm delivers better performance regarding layout. Laven, Leishman & Roweis (2005) examined the effectiveness of analytical model recognition algorithms for determining the difficulties of physical and logical layout analysis. They proposed these problems in the quality of heuristic, grammar-based, and rule-based, approaches. and grammar-based techniques. They have developed a dataset (http://jmlr.csail.mit.edu) which contains $932$ page images from academic journals. Finally, they applied three statistical classifiers to the logical layout analysis problem. They obtained an average precision of $85.5$% among $16$ sections and a general efficiency of $86$%.

Antonacopoulos, Karatzas & Bridson (2006) proposed and discussed several essential problems circling the ground truth toward the performance evaluation regarding layout analysis techniques. They converged on some various steps like formation, description and creation steps in the setting of a unique dataset formed by the authors. The current dataset (http://www.prima.cse.salford.ac.uk/dataset) has been made freely available to researchers. Shafait (2008) modified the Breuel (Roman script text-line model) (Breuel, 2001) system for layout analysis to Nastaliq script, and formed a text line pattern for Urdu text-lines. For the evaluation performance of the specified layout analysis system, they scanned $25$ Urdu documents from various sources, which were classified into five ( $5$) classes i.e., magazine, book, poetry, digest, book, poetry, and newspaper. In the dataset, there are five images of each class. Their dataset (http://www.iupr.org/demosdownloads) has been made publicly available. The produced system was examined with data from chosen classes, and $92$% precision was achieved for book, magazine, and poetry documents due to approximately extensive inter-line spacing. Despite this, the accuracy reduces to $80$% for digest document level through small inter-line spacing and appearance of itemized lists, and for newspapers, the correctness falls up to $72$% due to many font sizes, inverted text, and defective quality of the page.

Smith (2009) introduced a different hybrid page layout analysis algorithm. The stated algorithm used bottom-up techniques to create an initial data-type position and find the tab-stops that remained applied while the page was formatted. For the detection of the column layout of the page. They used tab-stops to force formation including a reading sequence on identifying regions, and the column layout was applied in a top-down manner. Their achieved accuracy rate of the developed method is $92$%. The complete source code for their proposed work is freely available (https://github.com/tesseract-ocr) as part of the Tesseract.

Bukhari, Shafait & Breuel (2011) introduced a method for the layout analysis of Arabic and Urdu document images. These images comprised of a mixture of separate and multi-column layouts. They used a proper combination of well-accepted, hardy text and non-text segmentation, text-line extraction, and reading sequence perception methods in the given layout analysis system. They assessed the presented system on $20$ Urdu and $25$ Arabic document images having different layouts. They achieved accuracy toward textual data and non-text segmentation was $99$% for the Arabic dataset. For text-line extraction, they used ridge based method and achieved accuracy of $96$% for the Arabic dataset while $92$% for the Urdu dataset. Erkilinc et al. (2011) introduced an algorithm for page layout analysis and document classification. In their proposed algorithm photo, text, and line/solid boundary areas classified. To improve and evaluate the proposed technique different varieties of easy, difficult, color, and gray-scale documents were applied. They made use of a text detection module and Run Length Encoding technique (RLE) (Xu et al., 2004), and a second module to identify photo/image and illustrated sections in that input document. Behind that, to get out photo candidate domains they worked a block-based classifier applying basis vector predictions. By using the Hough transform and edge-linkages analysis, the ultimate module recognizes lines and sharp edges sequentially. The photo, text, and strong edge/line maps are composed to create a page layout analysis of the scanned objective document. The investigation outcome shows $85$% accuracy on average for the introduced method.

Tran, Na & Kim (2015) proposed a method for textual and non-textual segments classification in document images. Their introduced method is the combination of whitespace analysis method with multi-layer homogeneous regions which are called recursive filters. Empirical outcomes on the ICDAR $2009$ page segmentation competition dataset and other datasets illustrate the effectiveness and advantage of the intended approach. They obtained above $90$% efficiency for text detection, no-text detection, text region detection, and non-text region detection. Ahmad et al. (2017) presented a text-line extraction method toward the extraction regarding large titles and headings in Arabic like scripts. The proposed method is based on a Hanning window smoothing procedure and Horizontal Projection Pro-file (HPP) (Javed, Nagabhushan & Chaudhuri, 2014). They obtained an accuracy of $99.30$%. The weakness of their introduced approach is that it requires d-skewed images.

Motivation

It is a valid fact that the synthetic data does not meet all the real-world requirements that could cover the real challenges of layout analysis and classification. The classifiers trained on synthetic data have a deficiency in generalization towards real data. Hence, efficient investigation mainly relies on covering real data rather than synthetic data. But, concerning the Pashto language, we do not have any such real data that can be analyzed for document layout analysis. Accordingly, we need to produce a real dataset that comprises primary patterns/ layouts and other complexities present in the Pashto language. To accomplish this purpose, we have created a new dataset that contains real-world samples of camera captured images presenting an appropriate benchmark for the Pashto DIA system. Some representations of Pashto text and graphics are shown in Fig. 3. Certainly, this research not only provides the scientific domain but also benefits the research society regarding regional languages. On the other-side real data shows more challenges due to the formation of various stages in the process.

Further, to accomplish useful document layout analysis and classification, we require a suitable dataset toward our newly developed systems and models. Hence, the purpose of this section is to report the creation of a real dataset for the Pashto language in the domain of document layout analysis and classification.

Data acquisition

Data acquisition remains an essential stage in the dataset creation. The sources concerning data acquisition must be a scanner or a camera-based device. In usual, the scanner-based acquisition is admitted as a conventional method and is the most applied method towards the digitization of books and other historical stuff. Although, the scanner-based document is examined to be pretty simple due to their acquisition in a restrained condition. Moreover, you have to have an extra device named scanner. In contrast, modern growth in mobile technologies in the formation of smartphone gadgets, etc., the camera-based data acquisition brings notable concentration towards digitization.

It is to consider here that camera-based capturing frequently undergo due to artifacts like shadows, skew, perspective distortion, warping, light reflection, and blurriness. While the scanner based acquisition hardly presents skew and lack of different artifacts as compared with a camera-based acquisition. It indicates a classifier trained on camera-captured images will hold the strength to be extra robust associated to the one trained on scanned documents. Meanwhile in our research, we use a camera-based image acquisition procedure. We operated a mobile camera to captured the images of the chosen Pashto documents.

Dataset description

Our newly created dataset contain 1,017 Pashto document images captured by the camera. We named our dataset Camera Captured Pashto Document Image (CCPDI). The sources (https://www.taleem360.com/categories/text-books-kpk?page=2) for acquired books for the CCPDI dataset given in Table 1. To annotated the CCPDI dataset, we worked on a tool called labelMe. The annotation of every textual and graphics segment is made by analyzing its contour/edges by taking polygons. The concern annotation toward every image is saved in a separate $.json$ file.

Data annotation

In supervised learning, the learning is achieved by contributed input-output pairs. It means for each input we have to provide a respective class label (Hussain et al., 2022). In short, we have to transcribed/ annotated or make ground truth for our data. Accordingly, annotation is the method in which we give a label to data like text, videos, and image, (our case). The Fig. 4 presents the input labeled image with the corresponding ground-truth. This is performed by specifying some kind of keywords in the appropriate area of text, image, etc. Normally, this approach is used in the scope of AI and ML areas to train the machines. A necessary step in improving any computer vision model is to establish a training algorithm and approve certain models applying high-quality training data.

Neural network

A neural network also called an artificial neural network (ANN). The system of hardware/ software imitated after the performance of neurons in the animal brain. This is one method for formulating artificially intelligent programs. Neural networks are a model of machine learning, where a program can change as its weight learns to solve a problem.

Object detector: region-based object detector

The single shot detector

The single shot detector (SSD) is a variant of Faster R-CNN. It has no designated region proposal network. SSD divines the boundary boxes and the classes direct from feature maps in one single shot. Through our research purpose, we worked mainly on SSD (Liu et al., 2016) model. SSD is created for object detection in real-time. Faster R-CNN works on region proposal network to form boundary boxes and applies those boxes to classify objects. It is considered the state of the art in accuracy, the whole process runs at seven frames per second. SSD speeds up the process by decreasing the requirement for the region proposal network. To improve the drop in accuracy, SSD works many enhancements including multi-scale features and default boxes. These improvements enable SSD to match the Faster R-CNN’s efficiency applying lower resolution images, which further accelerates the speed higher (Liu et al., 2016).

SSD model

The SSD method provides a limited-size number of bounding boxes and rates concerning the appearance of object class occurrences in those boxes, served by a non-maximum suppression-step to perform the final detection. The initial network layers are based on a standard architecture that trained for high-quality image analysis, which we call the base network. Further, the auxiliary structure added to the network to achieve detection with the following key features (Liu et al., 2016).

• Multi-scale feature maps for detection.

• Convolutional predictors for detection.

• Default boxes and aspect ratios.

The SSD object detection is composed of two parts. First extract feature maps and then apply convolution filters to detect objects.

The SSD works VGG16¹ to extract feature maps, and next, it detects objects applying the Convolution in layers $4-3$. The SSD architecture is trained on more than a million images of the ImageNet database.² It calculates both the location and class scores utilizing small convolution filters. After extracting the feature maps, SSD uses $3\times 3$ Convolutional filters for every cell to perform predictions. Certain filters determine the results just like the regular CNN filters. The typical SSD model is shown in Fig. 5.

Unlike its predecessor R-CNN (Girshick, 2015), the SSD does not use a selected region proposal network, instead, it divines the boundary boxes toward the classes direct from feature maps in a single pass. This network is renowned for speed and performance regarding object detection problems. SSD uses the VGG16 network as a feature extractor. Next combine custom convolution layers (blue) afterward and apply convolution filters (green) to make predictions (Girshick, 2015; Liu et al., 2016).

However, convolution layers decrease spatial dimension and resolution. Therefore, Fig. 6 shows the detection of large objects only. To fix that, we perform autonomous object detentions from multiple feature maps. Figure 7 shows the multi-scale feature map for detection. The dimensions of feature maps is shown in Fig. 8. SSD uses deep-down layers to the convolutional network for the detection of objects. If we redraw the picture closer to scale, we should understand that spatial resolution has decreased significantly and may already drop the chance in locating small objects that are very difficult to detect in less resolution. If such difficulty exists, we require to enhance the resolution of the input image (Girshick, 2015; Canziani, Paszke & Culurciello, 2016; Liu et al., 2016).

Evaluation criteria

We will use mAP (mean average precision) and Intersection over Union (IoU) for the evaluation of our proposed model. The mentioned metrics are briefly explained below.

Precision and recall

Precision measures how much the prediction is efficient. i.e., the percentage of predictions are accurate.

Recall measures how useful you achieve all the positives. The mAP can be calculated via Eq. (1).

(1) $$mAP=\frac{{\sum }_{q=1}^{Q}avP(q)}{Q}$$where avP (q) is the average precision (AP) for a given query (q) and ‘Q’ is the total number of queries.

Data split

In supervised learning, proper train sets, as well as test sets, are extremely essential for achieving robust classifiers. For this purpose, we have split the dataset (shown in “Experiment”). We have worked a hold out method, in which about $70$% of data belongs to train-set and about $30$% goes to test-set. We particularly used the same proportions. However, before utilizing this split, we have mix-up the entire dataset first. The total 1,017 images are split in a ratio of 71:29. We could get a train-set of $71$% images and a test-set of $294$% images. The corresponding ground-truth data are similarly split according to the images.

Experimental procedure

The empirical procedure has been divided into two sub tasks. The first task is Training the model and the other task is exporting the model. To accomplish these two sub tasks the split data has been converted to TFRecord file format. TFRecord file is the combination of the ground-truth and the corresponding image file formats.

Training the model

After the creation of the input file for API, the selected model has been trained by fulfilling the following specifications.

• An object detection training pipeline. They also provide sample configuration files on the repository. The SSD-mobilenet-v1-config has been utilized as a basis in my training. The need for the adjustment of the num-classes to two (Pashtotext, graphic) has been covered. I worked the default settings in terms of other configurations like the learning rate $0.003$, batch size $24$.

• The dataset (TFRecord files) and its corresponding label map. The label map file includes the class names, it is needed to the classifier to learn textual blocks corresponding to class labels. In our case, the file has only two class names i.e., “Pashtotext” and “graphic”. Figure 9 also shows the snapshot of the label-map file.

• Pre-trained model checkpoint. A pre-trained model is used to train the API for our own task. Because for limited data a completely un-trained model will take large time to converge. This process is also known as re-tuning or fine-tuning an already available models. We use SSD-mobilenet-v1, which is already trained on Pet dataset. A pre-tuned model the SSD-mobilenet-v1 as this model speed is more important than accuracy has been used.

The training process has been performed locally. While training the evaluation job also done after every 100 epochs. By running Tensor-Board on our machine the training and evaluation process has also been monitored. The training ran over $29.75\mathrm{k}$ steps with a batch size of $24$ and achieved good results in about $11\mathrm{h}$.

Machine platform

The experiments have been carried out using a machine having the following hardware configuration; Intel Core $(TM)i7$, $2.8\mathrm{G}\mathrm{H}\mathrm{z}$, with $16$ GB RAM. The machine has a GPU of Nvidia GTX 1,070 chip, with $8$ GB internal memory.

Results

After 29,750 epochs, we have stopped the training, and the final model was selected for evaluation. The evaluation was done by examining the $300$ images from the test set. In this way, We achieved a mean average precision (mAP) $84.90\mathrm{\%}$. Figure 10 shows the mAP in detail along with corresponding epochs as well as total elapsed time.

Similarly, the training process was tested for how it reduces the total loss. The total loss that we finally achieved is $4.6$, while its ceilings were approximately $8.9$. Figure 11 depicts the value of loss with regard to epochs in the training.

Discussion

After an accurate analysis of results as well as individual images, we have important findings that are described in the following lines. Before we could empirically asses the results, Fig. 12 shows some prediction of our proposed model on totally unseen data. To know the visual description of our examples, please understand the notions mentioned here. The light green color for the bounding box represents “Pashto text” and the light blue color represents “graphic”. The first row in the Fig. 12 represents the two cases where our model has shown the best performance. The image at left shows the prediction and the image at right shows the ground-truth.

On the other hand, the middle row describes images with less complexity which is shown in Fig. 12. The main cause is the quality of the bounding boxes, for example, some bounding-boxes partially bound Pashto text as well as the graphic. In general, our proposed method has proven better performance while checking it via human eyes.

Although, the final row in Fig. 12 presents some images with less detection and miss-classification. I thoroughly examine the reason, and I found that for small bounding-boxes the prediction is not good as for large bounding-boxes. Additional investigation yields that the instances for the small textual blocks are similarly less than the large textual blocks. This non-uniform distribution affects the classifier to determine more about large textual blocks rather than small textual blocks. I further validate this point by specifically considering the mAP for only small bounding-boxes, and therefore it is proved there that the mAP for small bounding boxes is just $25$%. It can be observed in Fig. 12.