How much energy is needed to build a solar panel?

First you couldn't put a fully automated plant in the desert - much is automated but it requires lots of people and industrial infrastructure close at hand.  There are lots of chemicals, energy sources, technical talent, etc. that has to be there to keep the supply chains filled, run things and fix things. All of these means you'd never build in such a location.  Instead you build the fabs where you have fabs already.  Today that is Taiwan and China.  You can NOT magically ignore these as they are as critical as having a supply of silicon or energy. But if you insist you can estimate the question specifically - simply calculate the reaction energy of reducing SiO2 to Si (basic high school chemistry) as the minimum energy required from somewhere per unit weight of silicon. Then multiply by the average about of silicon required per collected watt of solar energy. Maybe sunlight could supply the manufacturing energy in theory but practically incident sunlight isn't concentrated enough as a supply for that kind of chemical reaction - it's about thermodynamics and the realities of the Earth orbit around the Sun).  And even this energy would still a tiny fraction reaction for all the things you insist on leaving out but would still be necessary.  You are abstracting the problem to a level that tells you nothing.