Motivated by the need to solve a real-world application, we propose a novel model for extracting relationships in tasks where the label space is large but can be factored and the training data is limited. The model tackles the problem in multiple stages but is trained end-to-end using curriculum learning. Each stage realizes simple intuitions for improving the model and through ablation analysis we see the benefits of each stage. We evaluate our models on two tasks, that of extracting symptoms and medications along with their properties from clinical conversations. While LSTM-based baselines achieve a F1-score of 0.08 and 0.35 for symptoms and medications respectively, our models achieve a performance of 0.56 and 0.43 respectively.