Conventional image outpainting methods usually treat unobserved areas as unknown and extend the scene only in terms of semantic consistency, thus overlooking the hidden information in shadows cast by unobserved areas, such as the invisible shapes and semantics.In this paper, we propose to extract and utilize the hidden information of unobserved areas from their shadows to enhance image outpainting.To this end, we propose an end-to-end deep approach that explicitly looks into the shadows within the image.Specifically, we extract shadows from the input image and identify instance-level shadow regions cast by the unobserved areas.Then, the instance-level shadow representations are concatenated to predict the scene layout of each unobserved instance and outpaint the unobserved areas.Finally, two discriminators are implemented to enhance alignment between the extended semantics and their shadows. In the experiments, we show that our proposed approach provides complementary cues for outpainting and achieves considerable improvement on all datasets by adopting our approach as a plug-in module.