Cusco is one of the most beautiful cities that we've visited on our travels around South America. The historic centre was like an open air museum of colonial architecture, a warren of narrow, cobbled streets that all seemed to lead back to the majestic Plaza de Armas.

But there was also a lingering sadness to Cusco. The city was once the capital of the Inca empire, said to have been founded by the first Inca emperor, Manco Capac. When the Spanish conquistadors arrived in Cusco they were impressed by the exquisite stonework, and astonished by the temples gleaming with gold. What followed was perhaps the greatest act of cultural vandalism of the whole Spanish conquest.

We wanted to gain a better understanding of the history of Cusco, so we signed up for a free walking tour with a company called Inkan Milky Way Tours. Our guide, Richard, set up the company with his brother Elvis (who we later met in Lima), and he gave us a quick primer on Peruvian history, from the pre-Inca period up to the present day. The Incas were by no means the first great civilisation to emerge in this part of the world; Peru was home to the Nazcas, the Moches, the Chimus and the Huaris, to name just a few.

The golden age of the Inca empire was actually pretty short, lasting just 100 years or so. From small beginnings the Incas exploded out of the Andes during a period of rapid expansion, bringing many neighbouring empires and civilisations under their rule. At its height the Inca empire, known in the Quechua language as Tawantinsuyu, stretched from southern Colombia down to northern Chile, and from the Pacific Ocean to the edges of the Amazon rainforest.

When the Spanish conquistadors arrived in Peru in 1532, under the command of Francisco Pizarro, the Inca empire had been devastated by a civil war between two competing claimants to the Inca throne: Huáscar, whose power base lay in Cusco, and his brother Atahualpa, who controlled Quito in the north.

These divisions, combined with the devastating introduction of diseases such as smallpox, meant that the Inca empire was ripe for the taking. There was also a huge disparity in military technology; the spears, arrows and clubs of the Incas stood no chance against the horses, cannon and muskets of the Spanish.

Once the Spaniards had defeated the Incas and taken control of Cusco, they began dismantling the Inca palaces and temples, building their own city on top of them. You can clearly see this today; many of Cusco's colonial houses and churches sit on the remains of Inca walls, which are easy to identify thanks to their amazing stonework.

The Incas perfected a style of architecture in which the stones are precisely cut and slotted together, with no mortar between the bricks. Inca buildings also had sloping, trapezoid walls, which helped protect them against the earthquakes that are common in this region.

Although Cusco's colonial architecture is impressive, it seemed such a shame to me that the Spanish chose to dismantle the Inca city and rebuild it in a European style. There are still tantalising glimpses of what Cusco would have been like; many of the narrow streets in the old town follow the old Inca street plan, based on the shape of a puma, and are still flanked by Inca walls. It is only in recent years that Cusco has started to celebrate its Inca heritage; in the centre of the Plaza de Armas is a statue of Manco Capac, erected in 2011, a reminder that this colonial square was once at the heart of the Inca empire.

Modern Cusco is also one of the most touristy cities in South America, thanks to its proximity to Machu Picchu. This brings a level of hassle that we hadn't really encountered anywhere else, with people constantly trying to sell us a tour or a massage, though the touts were a lot less persistent than in other parts of the world.

Cusco is a great city for just wandering around, and we were able to enjoy luxuries that were often lacking in Bolivia, such as a comfortable bed, reliable hot showers and a varied choice of restaurants. We spent three days in Cusco preparing for our Inca Trail hike, and another three days recovering afterwards, so we got to know the place pretty well by the end.

Perhaps the most stark illustration of the rupture caused by the arrival of the Spaniards is the church and convent of Santo Domingo, which was built over the top of the Qorikancha, the Inca sun temple. This was the most important temple in the Inca empire, with towering stone walls and an interior glistening with gold. You can still see the huge, perfectly cut stones in the foundations of the church, and the same sort of distinctive curved wall that you see at the sun temple at Machu Picchu.

We wandered around the church looking at the elaborate colonial interiors and the collection of Catholic religious art, but the most interesting part of the complex was the surviving Inca architecture in the central courtyard. These structures were built over by the Spanish and forgotten about, but were later rediscovered after two large earthquakes caused the newer buildings to collapse and revealed the sturdy Inca walls still standing.

On our last day in Cusco we walked through the narrow lanes of San Blas, a picturesque barrio in the north of the city, and up some steep steps to the site of one of the most famous battles in the city's history. It was here in 1536 that the Spanish mounted a sustained and bloody assault on the fortress of Saqsaywaman, eventually overcoming Inca resistance and breaking a 10-month siege that had been slowly strangling Spanish-held Cusco.

For some reason, many of Cusco's most important archaeological sites can only be accessed by buying a ticket called the boleto turístico, which covers entrance to 16 different sites and costs 130 soles (about £31) for the full 10-day ticket, or 70 soles (about £17) for a 1-day ticket that only includes some of the sites. It's all well and good if you want to visit lots of different sites, but if you only want to visit one, as we did, then it becomes a total rip-off, especially by South American standards. We found this pretty frustrating, as we would have happily paid, say, £6 just to see Saqsaywaman, but this wasn't an option.

We were on a strict backpacker budget, so we decided to save our money and walked around the perimeter of the site instead. Luckily there are several points where you can see over the hedge into the fortress, and we had a decent view of the huge grey stones that the Incas dug into the hillside. The ruins were one of the most impressive Inca constructions that we saw in Cusco, despite the fact that we had to gaze at them from a distance.

After completing our circuit we hiked up a small hill to a statue of Jesus overlooking the city, like a miniature version of Rio's Christ the Redeemer. The views from the top were fantastic, with the terracotta tiled roofs of the colonial city laid out beneath us. It was a scene that the builders of Saqsaywaman would not have recognised; from this distance, Cusco's Inca foundations were completely hidden from view.