Back in 2006 HBO aired a documentary called Boffo! Tinseltown's Bombs and Blockbusters. One of the people featured in the doc was George Clooney. I wish I could find the exact quote, but Mr. Clooney said something to the effect that no one in Hollywood knows how to make a hit movie because if they did they would do it all of the time. He added that the best they can do is hire the people they believe are best for each job, film the movie, release it, and hope it finds an audience.
Taking this a step further, I'd say those same people are more often than not equally clueless about why a movie succeeds or fails even after audiences make that determination. They might think they know, so they take this and that from the first movie and include those elements in the sequel, and that's why so many sequels are watered-down versions of the original movie. And, of course, there's that "top that" mentality that MikeKardec mentioned above. "Now that we know what works, we can do it better!" Wrong.
No question about that. The quote a la Clooney is a paraphrase of the famous observation in Adventures in the Screen Trade by William Goldman - "Nobody knows anything". An entire chapter of Goldman explores this idea.
But really it only partly applies to the Indy films (maybe to 4). They continued to have hits with the Indy films, so why would anyone care if a few people quibble over the dodgy scripts and bad effects? Last Crusade was an enormous hit and many like it as much as Raiders. So in actual fact they knew what they were doing.
For me the sequels failed as artistic works - as good screenplays and film making. And there are many who would agree with that. But this never effected the bank accounts of messes Lucas and Spielberg.
What I find fascinating is how often a hit is an artistic failure and a flop is an artistic triumph.