I've always wondered why some people don't like musicals. My speculation is that they don't believe music can be used to tell stories. Maybe they don't really like singing at all. Maybe they only listen to music in the background or in short bursts on the radio. If you're used to two minute pop songs, a two hour musical is a long thing to sit through.
There have been many articles written about the decline of Broadway. Between 1957 and 1967, 8 of the best-selling albums in the US were cast albums for musicals like My Fair Lady and West Side Story. Those days are over and that style of music is just not mainstream anymore. Musical theatre is a niche art, and Broadway is a tourist attraction. The reality is that Broadway will support itself with big commercial hits and do some challenging "indie" stuff on the side (like next to normal.)
Writers will need to decide whether their musical to be popular (Hollywood blockbuster) or to be art (independent film) . If you decide to do art, forget about those people who never liked musical theatre in the first place.
In this difficult climate, should people keep trying to write the next The Light in the Piazza or next to normal?
It's extremely difficult to write dramatic musical theatre. It's much easier to do a comedy or write an ironic musical. If you do a sincere drama, you're fighting against the fact that music is a broad, heavy brush to paint with. For whatever reason, society is not very music literate. Mainstream people don't sit around debating Mahler. Maybe you can tell interesting stories with Wagnerian motifs and complicated harmonies. The number of people who would get that (people who study classical music) are even smaller than the number of theatre fans. So you're stuck with traditional Broadway style and rough genres of music like jazz, gospel, rock, pop, folk, etc. These styles of music are associated with a particular mood (think of Wong Kar-wai films) as opposed to specific statements. We don't have a musical equivalent of dialogue (or maybe there is, but it's beyond my poor, poor musical knowledge?).
Traditionally, music is used in theatre only when the characters experience extreme emotions such that they can no longer speak and have to burst into song. Traditionally, the audience expects the actors to sing on the beat and in tune. These things are limiting, compared to dialogue which can be said with any kind of pitch, emphasis, or timing as long as the words get across. There's no reason why people can't write more musicals where music is played continuously. I liked the film-like underscoring in next to normal. Operas and many British musicals (e.g. Les Miserables) are sung-through. Singing off-beat and out-of-tune would be much harder. People mostly expect beautiful singing in musical theatre and that's a tough expectation to break.
The key to writing dramatic musical theatre is to find or write a story where you can figure out a way to add music in a way that enhances the work rather than being an adornment. It's hard enough to write a good drama in a 300 page novel where you have space and thousands of words. How many stories feel like they're crying out for a musical treatment? My guess is not many. I think the odds are really stacked against the writers. Musical theatre tends to be most successful when the music is trying to be funny, cute, or beautiful.
Still, there's no reason why people shouldn't try. If enough people like it and are affected by it, the work was worthwhile. Everyone should be able to find a piece of art that speaks to them, that helps them understand the world or themselves better, that is cathartic and helps them express their feelings. If your cup of tea is musical theater (like me), why not? If someone wants to paint the equivalent of Hamlet, why not?
I never thought of musical theatre as serious art, but then I studied Sondheim and encountered next to normal, which inspired my appreciation and understanding of theatre, acting, and music. I'm truly grateful to the people who push the boundaries of musical theatre and I'm hoping for more. Finding inspirational art is a rare and special experience, not unlike falling in love. Everyone should "fall in love" with art at least once.
(I am not an expert on musical theatre or music. I'm just writing my random thought for the day.)