Our driver Majid stood in deep conversation with a man with a long beard, flowing robes, and fierce eyes. I’m sure he had mountain warrior blood in his veins. This is Oman, where the khanjar, the dagger is standard ceremonial wear, and the shotgun a mark of honour.

Avril and I stood apart, awkwardly aware that we were a novelty in this man’s world. Tourists. The stone houses clinging to the slope, and steep, dark alleys with steps smoothed by the centuries, were remote until recently. Now a very expensive road, funded from oil riches, came sashaying up from the valley below.

I kept my gaze on some nearby mountains, a red-grey jumble of crazy contortions. An eagle drifted in a thermal. (I couldn’t work out which of four possible species of eagle it was in this bird-rich land, when I looked it up later.)

More talking. Then, abruptly, Majid left the villager and rejoined us. I assumed he would pass on a polite translation of something rather warlike, like: ‘He says push off. And don’t return.’ I’d be annoyed too, if outsiders invaded my village. But even to think such a thing maligns Oman, so deep do ancient courtesies and kindness run here.

What he actually said was: ‘Oh, the man asks you to please come to his house, take coffee, and be his guest. I thanked him but said you were in a hurry to see some more forts before dark.’

This from a perfect stranger. Around the world it’s a human instinct to power up the tills when tourists arrive. But in Misfat al A’Briyeen nobody even tried to sell us a postcard. These people were offering nothing but Oman’s famous hospitality.

It was the second surprise that day. The first one was that, only two hours from the frenzy of Dubai, such a glorious emerald outburst even exists in the Arabian peninsula. Water burbled cool and pure in a flat gravity-fed channel, the falaj, which corkscrewed through the village. We were about 90 minutes from the capital Muscat. But take out the palm trees, and we could have been in a village in the Alpujarra, the mountainous region in Southern Spain.

Oman is on the cape on the NE corner of the Arabian peninsula, close to Dubai, Abu Dhabi and the other United Arab Emirates (UAE) kingdoms. It is much bigger than its near neighbours, and with many more natural wonders to build its still small tourism industry on.

600 miles of scintillating coast and beach is as undeveloped as the Costa Brava must have been in about 1910. Inland Oman is a power landscape of towering mountains scattered with deserts, lush oasis settlements – wadis — and a surprisingly green south country. It has its own Grand Canyon. But the pride and joy are the 1000 or so watchtowers, forts and castles. They are restoring many of them, again with oil money. Our trip into the mountains was to see three of the best, with the detour to this village as Majid’s surprise bonus.

Like the UAE kingdoms, Oman is a stable and peaceful destination in the big fractured muddle of the Middle East. But while soaraway Dubai and Abu may be of the hottest holiday tickets around, you come here to recover. As they build towards the stratosphere, Oman don’t even have skyscrapers. And it is spared the prodigious gridlock now frequent on the near neighbours’ roads.

We landed in Muscat at midnight. The Shangri-La hotel group, keen to match the Dubai glitz, lays on a Bentley for guests at the Al Husn (the castle)  hotel, part of its Barr Al Jissah  resort. We sped past immaculately tended flowerbeds lining the road from the airport. By now, other countries would be talking big. For example, the four storey high, in your face image of the leader. Not in Oman. Sultan Qaboos, in charge since 1970, is a small portrait man.

Instead of skyscrapers, the big impact buildings were the blue and gold mosques, each in its square of four pencil- thin floodlit minarets. And an illuminated giant frankincense burner and coffee pot, celebrating vital details in the Omani home.

The dashboard GPS showed us leaving Muscat city for a cartographical black hole, an utter blank. We were no wiser looking out. We were on a steep and clearly very expensive road, blasted through a mini range of bare, brooding mountains. We reached the resort’s security gate, and it all made sense. They’d hidden it on a private headland behind a big mountain barricade. If I was the sort of person the paparazzi could be bothered to tail, this is where I’d come to dodge them.

The Al Husn treats arriving guests to a crash course in Oman’s culture and architecture, in its decor. We signed in in a cool marble foyer, under horseshoe arches lined with gilt and giant bronze palm trees. A flock of inanimate flamingos browsed in a mosaic pond. The influences were Moghul, Moorish, Chinese and Arabic, bundled up to create the Omani style. There was a pervading aroma of frankincense. It’s Oman special smell: the tree they tap it from grows in the green, abundant land around Salala, in the south.

The man from reception led us down long carpeted corridors, lined with photos and paintings depicting Oman’s man made and natural wonders — headlands, mountains, castles and forts — alongside framed national costumes. But the 50 carat moment came when he opened our balcony door onto a silver sensation, the full moon speckling the Indian Ocean as it crashed onto rocks below.

In the morning sun our hotel did look very much like a castle. We wandered under its pink “ramparts”, investigating its cool marble enclaves, testing the infinity pool, and reading the menus at the resort’s eight restaurants.

Big, bold developments like this are bound to dent the environment a bit. Laying this important foundation stone in Oman’s tourism drive involved major rearranging of some mountain habitat. But Shangri-La is claiming environmental pay back. It has moved a coral reef –time will tell how successful that has been.

And it is actively protecting nesting green and hawksbill turtles on a beach within the resort. Guests can observe the turtles, from a respectful distance, guided by the hotel’s resident turtle ranger. He’s very pleased with progress so far, with a 70% increase in the number of nesting females in 2007, compared to 2006.

We skipped free tea and cakes at 4 pm in the lounge, among the great decorated vases and sumptuous carpets, and took the free shuttle bus to the Mutrah Souq in Muscat, ten minutes away. Muscat has been a famous centre of trade for centuries. The adventurous Omanis were fine seafarers, riding the monsoon winds to far-flung lands. The Portuguese took control in the 1600s, followed by the Turks. Since then the British have been good friends to an independent Oman. The words “HMS Falmouth” are etched into the hill above the old port, commemorating the British sloop which sank an Italian submarine close to here in 1940.

In no time we were walking the souk’s tight and narrow ways. Crammed together stalls and shops displayed the staples of the Arabian market. In no particular order there was gold, bread, T-shirts and jeans, hijabs, spices, lamps – some, I’m sure, containing a genii — shoes and stands for holding the Koran. And that speciality aroma, frankincense, was everywhere.

Two British ladies were lugging an ancient chest. If only they had know. Men sit by wheelbarrows in the centre of the market, ready to push bulky items to all parts. All part of the souk’s ancient service culture.

A short step away was the Corniche, the waterside drive lined with former merchants houses picked out in colourful lights. Across the road, on a wide walkway, a long line of men worked the waters for sardines with hand lines, occasionally sidestepping  the breaking  waves.

We joined the sunset fashion parade. The men wore the dishdasha, a simple, ankle-length, collarless gown. Most were white, but some in black, blue, brown and lilac, with subtle variations in pattern and style to denote their tribe.

But it was the headgear that turned our heads on this catwalk. The mussar starts as a finely woven coloured woollen or cotton scarf, which men wear around their necks to keep out the night cold. Then the skill begins. With much dexterity they weave it tightly into a turban around their intricately embroidered caps, as a proud emblem of their clan.

And fashion isn’t only for the men. We saw possees of teenage girls in black chadors. (We counted only two women covering their faces on our entire trip). Perfectly in control of their style, they hunted in packs in the malls and the souks, stopping now and then to adjust their makeup. And check their mobile phones.

Next day at the hotel, reception produced a fat book of outings. We took the standard desert day trip. Marvelling at a vast, golden expanse of sand from a comfort of a Land Cruiser, you grasp the enormous heroism of those non-mechanized desert crossers such as Wilfred Thesisger. He set off from Salalah in southern Oman on his epic crossing of the Arabian empty quarter in the 1930s.

Today desert travel is a doddle,  But drivers still like to test your nerves, by letting the vehicle drift down a near sheer 200 feet high dune. Eventually you feel you have reached the point where the vehicle must tip over. But it doesn’t, and you slide slowly to the bottom with emotions of mighty relief.  And whiter than white knuckles.

But adjoining countries do deserts too. Oman’s unique attraction is the day trip into those beautiful and austere grey-red mountains to see the ancient citadels. That was our second day out, with Majid, to see Rustaq, Nizwan and Abreen, every bit as solid and imposing as Caernarfon and Conwy castles.

We inspected decorated ceilings, and cool carpeted interiors strewn with cushions, then looked down over crenellated battlements onto neat white homes and date palm plantations. Majid gleefully pointed to the murder holes through which defenders could pour hot oil or syrup onto invaders who somehow managed to breach those formidable defences. And everywhere that strange determination not to profit from tourists. We bought an excellent locally handmade pot for incense burning, and a postcard, for £1.30

Out of a blur of whimsical details on that fast trip to the forts, I remember the doleful look a camel gave us as it sped past, safely tied down in the back of a pickup. Then that first sight of Nizwa, the ancient capital, “Pearl of Islam”,
edging out from behind a crocodile-shaped line of hills. That fly-past by a flock of delicate doves, each with a flash of maroon around its neck. A shepherd dressed head to foot in red, striding across an ochre hillside. And the white egrets that suddenly alighted on the green grass of a roundabout in Muscat.

And so back to the hotel, where we narrowed the choice to the Moroccan restaurant, Shahrazad. Here I found my dish of the holiday, Pastilla Bil Hamam, a pie traditionally served at weddings, filled with pigeon meat, crushed almonds, scrambled eggs and covered with cinnamon and sugar.
Later, tipped off by a British couple in the lift, we headed for the Piano Bar for a nightcap, and live music from the resident pianist. (You should celebrate frugally, as alcohol is expensive in Oman.) The instant we sat down, a waitress carpet bombed our senses with strawberries dipped in chocolate. Somewhere far below the moon shone on the Indian Ocean as it crashed at the foot of the castle.

Fact Box:
I stayed at the Barr al Jissah resort, courtesy of Shangri-La Hotels,
www.shangri-la.com. 00 968 2477 6666. He flew to Muscat with Etihad, www.etihadairways.com. Fares start around £ 357, (via Abu Dhabi.  www.shangri-la.com. 00 968 2477 6666