Art and Science - When I say art, I mean writing by pure musical instinct, for the lack of better words. When I say science, I'm mostly referring to music theory. As you might know, you can write music to some degree using strictly theory. Whether or not the music will be stimulating or not is a different story. Even people who don't technically know music theory sometimes use it and don't even know they are. Theory is sometimes learned through experience and if you learned every Beatles song by ear and sort of figured out how they write, without knowing it you would be learning about secondary dominant chords and borrowed chords. Even though they didn't technically know in theoretic terms what they were doing, they were using a very standard musical theory that they certainly used by experience. I use both aspects to some degree when writing and depending on the song, this ratio of art and science changes somewhat.
Templates - There is also something that I call templates. This kind of music, or at least the chord progression is pretty much pre-determined. Two typical examples would be the "Blues" and "Rhythm Changes." There are variations on both, like a Minor Blues, Eight Bar or Jazz Blues. Rhythm Changes also have a pre-determined set of changes and the variations on these changes include various substitutions. For the most part, composers might use the "template" progression as is, and compose a melody over the top. Anybody who plays the Blues, knows about this.
Learning From Others - Some people will tell you that you shouldn't learn from other people's compositions because you'll never get an original sound of your own. I say that is the the stupidest advice I have ever heard. Out of all the great writers that I know, they all, and I mean every one of them learned through learning other composers music first. Whether they conceptualized everything using theory or not is a different story but regardless, they internalized certain tricks of the trade. So my best advice for you is to learn as many songs as you can and try to figure out how they came up with their hooks.
Me Personally - Since you are listening to "Prospects" let's see if I can't give you some background on how I came up with some of the songs. The methods may surprise you to some extent. The opening song, "Prospects" was basically an exercise or at least an assignment by my rhythm guitar teacher in college. I won't get into the details but it is all based on theory. I added in a little musical sense and made it musical. I basically wanted to do something lydian and that was my motivation. Chart >>>
"When Love Greets You" - the second song isn't based on any sort of traditional theory but I was able to write it because I played so many Wayne Shorter and Herbie Hancock songs over the years. There isn't much functional diatonic harmony going on but I learned from his music how to use 9ths and 7ths as melody notes. It is really pure influence so if I never played Wayne's songs from the Real Book, I could have never written a song like this (not to say that I can write anything as well as Wayne Shorter). Chart >>>
"Extraordinaire" - The next song is what I previously called a "Template Song." It is mostly based on a minor blues. And I couldn't have possibly written this song without having played a million other minor blues, like "Equinox" or "Goodbye Porkpie Hat" before hand.
So you can see my experiences helped me write all the music. I personally don't believe you can learn to be a musician in a musical vacuum so I suggest you copy and analyze as many songs as possible.