Make a habit of turning the key to ON but not immediately to START. Listen for the fuel pump to run for 2 seconds then stop. Probably need to turn off blowers & music, etc. It seems normal for pump to either run OK or not run at all.

Sounds like GiraldyR and Cheese have different problems.
GiraldyR might be battery cables, connections, or maybe contacts in the starter solenoid. Those would prevent (or delay) the starter from spinning the engine. Try the caveman test - hit the starter with a broomstick or something while trying.

Cheese says his engine cranks strong, but sometimes doesn't fire right away. How about fouled sparkplugs? Does it run real good after it starts?

There is occasional hesitation under acceleration. I'm thinking fuel filter or plugs. I'm hoping it isn't fuel pump.

I ended up with a new starter
 

I got too paranoid to go on and got the new starter. For the most part the issue appears resolved except that I have had to make a habit of turning the key to auxiliary and waiting for the fuel pump to engage before fully starting the car. If I don't I sometimes get a 'clunk' and then nothing. Then I try again and it starts up. :confused:

(funny I just now saw the post from Jim Blake - exactly like he said)

Update
 

Finally I was able to get some work done on the car this past weekend. Got some 40 F temp.. First thing I took my battery to autozone. It checked bad so got it swap thanks God still had warranty.(They no longer make good battery this days or what. I own this car for 9 year and 156k miles in NYC. I this will be my six battery. 4 oem and 2 duralast gold they only seems to last 1.5 year.) Back to story. So got new battery next morning same issue long delay.. So decided to order a recycle starter from ebay since this starter cost too much new 250 plus and car no longer is at age to be investing heavily..So swapped out this weekend Ban. Problem solve. car crank instantly all the time. I also open the old starter and checked the brushes, 3 out of 4 brushes were almost gone..So thank God I replace just in time. Probably will rebuild my old start get some new brushes and have it as spare.. Thanks

Just in case someone want to rebuild their honda 03-07 started here is the link for brusher on ebay Starter Brush Repair Kit Honda Accord 2 4L w at 03 05 | eBay

Is it a slow starter like the battery may not be fully charged? If so I had a similar issues. I chronicled it on my blog.

2003 Honda Accord Shade Tree