Over recent years, computer vision has made great strides towards annotating parts of an image with symbolic labels, such as object categories or segment types. However, we are still far from the ultimate goal of providing a semantic description of an image, such as "a man, walking a dog on a sidewalk, carrying a backpack". In this talk, I will describe some projects that use probabilistic models in an attempt to move us a little closer towards this goal. The first part of the talk will present methods that use a more holistic scene analysis to improve our performance at core tasks such as object detection, segmentation, or 3D reconstruction. The second part of the talk will focus on finer-grained modeling of object shape, so as to allow us to annotate images with descriptive labels related to the object shape, pose, or activity (e.g., is a cheetah running or standing). These vision tasks rely on novel algorithms for core problems in machine learning and probabilistic models, such as efficient algorithms for probabilistic correspondence, transfer learning across related object classes for learning from sparse data, and more.