The original site of Torres de Albanchez was on a nearby mound, defended by the castle of Yedra, already demolished, but of which some vestiges remain: remains of masonry walls, defensive structures and wells, dug out of the same rock. It is estimated that its origins may be in a Hispano-Visigothic community between the 5th and 7th centuries, which after leaving the Guadalimar plains settled in this strategic enclave. It was conquered by the troops of Fernando III el Santo in 1235, who granted it to the Military Order of Santiago, which promoted the construction of a new settlement in the current less abrupt place. Yedra Castle was declared an Asset of Cultural Interest in 1985.