YA defines the age group of the reader - 14-21 or so. Literary is the opposite of commercial. There's a good bet that if you read it in school it was considered literary, though it may have had commercial appeal or been considered commercial when written. This section of the forum contains some discussions on the difference.