The agricultural industry scenario in many industrialized countries has adopted an image processing system as a solution to automate the grading process in order to provide accurate, reliable, consistent and quantitative information in addition to the large volumes, which human graders are not able to perform. In Malaysia, the grading of palm oil Fresh Fruit Bunches (FFB) is still performed manually through visual inspection using the surface color as the main quality attribute. It is the intention here to introduce an automated grading system for palm oil FFB using a computer assisted photogrammetric methodology which correlate the surface color of fruit bunches, not the fruitlets, to their ripeness and eventually sorts the fruit to two predefined fruit categories. The methodology consists of five main phases, i.e. image acquisition, image pre-processing, image segmentation, calculation of color Digital Numbers (DN) (data manipulation) and finally the classification of ripeness. This computerized photogrammetric image processing technique using MATLAB® package which is integrated to a sorting system differs in various aspects from other digital imaging technique or machine vision system adopted for classifying fruit ripeness. A comprehensive discussion will be presented based on the results achieved through actual fruit testing on the prototype grading system. The main concern was to ensure the reliability of the computerized photogrammetric technique achievable and the system’s mechanism working as intended. The fruit classification ability of the system yields above 90% accuracy and taking not more than 25 seconds to classify and sort each fruit.