Rigid-body visual tracking is an active research field with many practical applications including visual surveillance and intelligent transport system. In this paper, we define a new problem domain, called visual growth tracking, to track different parts of an object that grow non-uniformly over space and time for application in image-based plant phenotyping. The paper introduces a novel method to detect and track each leaf of a plant for automated leaf stage monitoring. The method has four phases: optimal view selection, plant architecture determination, leaf tracking and generation of a leaf status report. The proposed method uses a graph theoretic approach to reliably detect and track individual leaves by overcoming the challenge of leaf-losses based on temporal image sequence analysis for automatically generating the leaf status report containing the following phenotypes, i.e., the emergence timing of each leaf, total number of leaves present at any time, the day on which a particular leaf stopped growing, and the length and relative growth rate of individual leaves. The proposed method demonstrates high accuracy in detecting leaves and tracking them through the early vegetative stages of maize plants based on experimental evaluation on a publicly available benchmark dataset.