Read the tags on plants or the descriptions on seed packs. Annual does not mean that the plant will live a year. It will last a season. Two examples are dill and basil. In south Louisiana dill and basil will bolt - flower and go to seed, in a matter of weeks. I plant a couple of plants each and plant a couple of seeds in the pot at the same time. I use large pots 14" - 16" for each plant. The pot can accommodate multiple seedlings.
Rosemary and thyme are perineal in my climate. Your winters are very harsh. You will probably need to move your herbs inside in a south or west facing window over winter. I don't see trying to grow herbs on a windowsill. They really need larger pots to flourish.
Anyone in the U.S. who wants to garden needs to make your County Agent your best friend. Every state has an Ag Center University. Every County has an Ag Agent and a Horticulture Agent. The Horticulture Agent works with home gardeners. Get to know your agent and ask for advice.
The service is free to you.