Essaouira has long drawn visitors from afar. There has been a settlement here, on this windy promontory on Morocco’s Atlantic coast, since ancient times, but it wasn’t until the 16th century, when the Portuguese built a fort to safeguard their trading routes between Africa and the Mediterranean, that the town’s fortunes began to grow. In 1760 Sultan Sidi Mohammed decided in a fit of pique to create a port to rival Agadir, further down the coast. He commissioned French architect Theodore Cornut to design the town, who constructed a symphony of blue and white villas joined by twisting cobblestone alleys with a broad sweep of promenade that curves around the bay.

This is a seaside town glittering with fishscales and feral with cats; they crouch in the doorways, stalk along the battlements and curl around the ankles of the tourists in the restaurants. Passing beneath the imposing Moorish archways leading into the medina you enter another age, into narrow streets where no motorised traffic is allowed, where hooded figures in djellabahs shuffle along like hobbits in yellow slippers. The main street is a jostle of vendors with carts piled high with fruit, donkeys tottering past under enormous loads and packs of lean young men in leather jackets flirting with girls in tight jeans and headscarfs.

Down towards the seafront we ate in a fish restaurant, perched on bright blue benches drawn up at long white tables. The fish was amazingly fresh, and had been landed that morning – grilled sardines were served with just a squeeze of lemon juice, there was red snapper, mullet, prawns, crab, langoustine and mussels. Afterwards, taking a break from wandering through the sunlit streets past shops that were closing for the siesta, we stopped at a café overlooking the square for a mint tea, served hot and sweet in a small pot crammed full of fresh mint leaves. Heading into the souks there were mounds of fragrant spices coerced into improbable pyramids, stacks of brightly-coloured pottery and the conical pots called ‘tagine’ which give their name to the national dish. At dusk we strolled along the battlements of Skala Fort in the sudden cool of evening, watching the sunset turn the waves golden as the sea pounded and roared at the walls.

The Rolling Stones came to Essaouira, soaking up the mellow feel of the town, and Cat Stevens was a regular visitor – still is, in fact. He discovered Allah in the town, renamed himself Yusuf Islam, and can sometimes be seen in residence at the Hotel des Iles. Jimi Hendrix was here too, and wrote ‘Castles in the Sand’ having been inspired by the setting. Today the only castle in the sand you’re likely to encounter is the gigantic toppled boulder at the end of the beach which resembles a fortress collapsing into the waves. It’s not quite the end of the beach though; if you had the energy you could keep going all the way to Mauritania and beyond. Buffetted by the wind off the Atlantic, it’s not really a beach for sunbathing, but the onshore breeze is popular with windsurfers, and Essaouira has catered for them in recent years by marketing itself as ‘Windy City Africa’. In the bay lies the island of Mogador, once used as a prison island like a North African Alcatraz. Prior to that it was famed for its shellfish; the shells were crushed and used as a dye even in Roman times.

Windswept and with sand in my hair, I was advised that I needed sprucing up a bit before dinner that night, and in one of the small streets off the main drag I found just the place for the job. Rachid’s barber shop was an oasis of crisp linen and trickling water, and I popped in for a trim and a shave. I was ceremonially swathed in white sheeting and Rachid painted cool, citrus-scented lather over my face in several layers. He then took a cut-throat razor, dipped it in meths, set fire to the blade to sterilize it, and began to sweep it over my chin with quick, deft strokes. After the first course it was time for more lather, and then a second pass of the blade, which I barely felt. He laid a hot towel over my face, then peeled it off and dusted me with scented talc. The face that emerged looked considerably fresher than the one that had walked in the door, and the total fee of this miracle of rejuvenation was 15 dirhams – one pound sterling.

That evening we headed to the Taros Café on the Rue de la Skala, set in a 200-year-old building which was once a merchant’s house, with whitewashed walls and bright blue shutters. Taros is Essaouira’s trendiest spot, and given the frequency with which the town acts as a location for Hollywood movies – Oliver Stone’s “Alexander” was filmed here, as was “Kingdom of Heaven” – you might just end up recognising the faces at the next table. We sat on the terrace looking out across the rooftops at the blackness of the sea where the lights of the fishing boats winked like distant planets, and knew that Essaouira had become a place we would return to.