One of the most popular quotations attributed to Einstein is: "Everything should be made as simple as possible, but not simpler." Indeed, it's one of my own favourites.
Unfortunately, I have never been able to verify its authenticity as something Einstein ever wrote or said. The closest I've come is in some words contained in an address by Einstein entitled "On the Method of Theoretical Physics" in The Hebert Spencer Lecture delivered at Oxford on the 10th of June, 1933:
It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.
Our experience up to date justifies us in feeling sure that in Nature is actualized the ideal of mathematical simplicity. It is my conviction that pure mathematical construction enables us to discover the concepts and the laws connecting them which give us the key to the understanding of the phenomena of Nature.
It is essential for our point of view that we can arrive at these constructions and the laws relating them one with another by adhering to the principle of searching for the mathematically simplest concepts and their connections. In the paucity of the mathematically existent simple field-types and of the relations between them, lies the justification for the theorist's hope that he may comprehend reality in its depths.