Vilches Castle is a fortified complex consisting of an outer walled enclosure with an alcázar standing on a large rocky outcrop inside it. The outer enclosure was originally constructed from rammed earth and mortar on a masonry base, which was reinforced during the Christian era with irregular masonry cladding cemented in place with mortar. The enclosure had a jagged, zig-zag layout that followed the line of the terrain, and was crowned with a series of square-shaped towers at strategic points. It also had a D-shaped gate tower that provided access to the interior of the complex. During the Christian era the castle was strengthened with the addition of various elements including a polygonal alcázar built of irregular masonry with mortar of lime and sand. On the east side of the alcázar there was a large tower with rounded corners that served as the castle’s keep. The interior of the alcázar was accessed via a bent entrance located on the south side of the structure. The corners of the castle may have been crowned by solid circular towers, of which one has survived along with part of a second (although not in its original position). These features enhanced the castle’s ability to defend itself against attacks by artillery and firearms. Subsequently, the alcázar underwent major alterations in order to convert it into the shrine of La Virgen del Castillo. 

Within this walled enclosure there is a series of structures from earlier periods that were reused during the Middle Ages. They include a cistern, which may have been built during the Ibero-Roman period and is approximately 9 metres deep. It is made of opus caementicium (Roman concrete), with a lining of opus signinum (a waterproof variant of Roman concrete). This period may also have witnessed the building of two rectangular structures, again made of opus caementicium, located at the centre and east end of the hilltop and whose functions are unknown. Following the Christian conquest a number of new structures were built inside the castle grounds, such as a small church (which may have been built on the site of an earlier mosque) dedicated to the apostle James. One building, which is now in ruins, was substantially modified in order to convert it into a farmhouse or barn, possibly in connection to the olive groves that covered the hilltop. 

Outside the castle walls there is a series of cave dwellings, which were dug into the slope and made into living spaces during the Low Roman Empire or Early Middle Ages. Many of them are located below the south wall and have remained in use to the present day.

The archaeological excavations carried out on the hill where Vilches Castle stands have revealed evidence of occupation during the Neolithic era, Ibero-Roman era and Low Roman Empire. The original fortifications may have been constructed during these periods and subsequently reused to build other defensive structures in the Middle Ages. 

Following the arrival of Muslims to the area, the site became home to a small walled settlement known as Hisn Bily or Baly, which was used as a refuge by the various rural communities who lived nearby and, at times of political instability, abandoned their usual homes in order to take up temporary residence inside the protective walls at the top of the hill. According to Islamic chroniclers, the Muslim population of Vilches bore witness to the war of succession between the future Emir Hisham and his older brother Suleiman Abu Ayyub, who was the governor of the kura (region) of Toledo and clashed with his brother’s troops near Hisn Bily. At the end of the 9th century, this area came under the control of the Muladí rebel Ubayd Allah ibn Umayya ibn al-Saliya, who ruled more than 100 hisns and hamlets within an extensive territory known as Sumuntan, which has been identified as the mountainous Sierra de Segura region. 

Between the 11th and 12th centuries, a frantic process of fortification took place as a result of the numerous conflicts that broke out in al-Andalus following the dismantling of the caliphate, exacerbated by the Castilians’ advance into the Sierra Morena mountains. Vilches was no exception, and strengthened and expanded its defences during this period. However, this did not prevent a coalition of Christian monarchs, under the command of King Alfonso VIII of Castile, from taking Vilches along with numerous other castles in the Sierra Morena region (such as Castro Ferral, Navas and Baños de la Encina) after defeating the army of the Almohad Caliph Abu Abd Allah al-Nasir at the Battle of Las Navas de Tolosa in 1212.

It was at this time that the original walled enclosure of the Muslim hisn was modified and a new defensive element in the form of the alcázar was added at the highest part of the hill. As the town’s population grew, more space was required, and so Vilches was extended out to its current location on the south-west slope of the hill.

According to some writers, following its conquest the town was given to the Castilian nobleman García Pérez de Vilches; shortly afterwards, however, King Ferdinand III included it as one of the hamlets assigned to the administrative district of Baeza, as recorded by the king himself in the land-grant certificates issued in 1231 and 1233, which show that Baeza was awarded several castles including that of Vilches. Subsequently, in 1272 and on the orders of King Alfonso X, Vilches attained a certain status and degree of independence from the city of Baeza, as it was granted the privilege of having an exclusive boundary or hamlet council. The same privilege was granted to the settlements of Baños and Linares. 

In 1627, during the reign of King Philip IV, Vilches was granted the status and privileges of a villa, which gave it control over its own affairs and definitively released it from the jurisdiction of the Council of Baeza.