How difficult is it to change from studying computer science to cognitive science?

Whether it will be difficult or not depends on you. Do you have any interest or talent in psychology or other fields like anthropology, sociology, or neurology? It's not necessary to be interested in all of those, but the more the better. If you're mainly interested in software engineering or computer science principles, it may be difficult to sustain your studies in areas like developmental or abnormal psychology, or to get enough grounding in biology to have a solid understanding of cellular and systemic level neurology.  OTOH if you can understand "cognition" as a process common to psychological, biological, and artificial systems, you may be able easily build bridges between these areas in your studies. (FWIW, I earned my undergraduate degree in cognitive science in 1985 -- back then I had to convince the college that it was a real field and get them to approve the requirements. My degree was a pretty even split between computer science, psychology, and biology.)

First off, let me preface anything I say by clarifying that I'm neither a computer scientist nor a cognitive scientist. I'm a graduate student in neuroscience. I have some familiarity with both fields though, and I'd be happy to answer your question. I'd just advise that you more heavily weight the advice of people actually educated or working in the fields that you're asking about. I would echo a point that made- any similarities between computer and cognitive science are probably going to be very metaphorical. They might be aspects that seem similar when you look at them in a particular way, but I doubt there's going to be much overlap in the precise "what" or "how" of your studies. But that doesn't directly address your question, I suppose. So let me say this- if you're comfortable thinking about problems in a very critical and scientific way; if you can compare ideas and analyze their relative strengths and weaknesses in answering a question; and if you feel you can consider both the big and small picture in an investigation, I think you would do great in cognitive science. Let me give you an example. Cognitive science is the study of how minds work, and so it's not impossible that a cognitive scientist might one day find themselves studying depression. In their background research, they come across the fact that women are roughly twice as likely as men to experience depression. Now why might that be? Maybe there is some biological agent that predisposes women to depression. Maybe  societal pressures are greater on women than on men. Maybe mental health professionals are more likely to diagnose women than men, even if both groups show the same symptoms. Who knows! So now, the researcher needs to come up with a hypothesis that can be answered by research. They need to understand that the results from their research is one tiny puzzle piece, and they should know how it does AND doesn't bring people closer to answering the big question. What are the problems with their theory? How can the change the theory to match all the knowledge on a topic? If you think you can get used to thinking like this, cognitive science could be a really good match. I wouldn't worry too much about knowing the specifics of psychology and neuroscience. The point of the program is to teach you those things. Yes, some schools will prefer to accept people that already have a strong background in those areas. And yes, some background knowledge will absolutely help you. I recommend getting an Intro to Psychology textbook, and Fundamentals of Human Neuropsychology (Kolb & Whishaw), and getting comfortable with some of the major ideas therein the summer before you start any program. However, I absolutely do not believe that a lack of in-depth background knowledge would preclude you from excelling any cognitive science programs.

I don't know about your program, but this was my (admittedly limited) experience at UCSD. I briefly switched from Computer Science to Cognitive Science because I was attracted to the HCI and social science components of it. Once I started taking classes, I realized there wasn't as much overlap between the two majors as I thought. It really is a broad interdisciplinary major, with a significant amount of psychology, sociology, and linguistics, for example. Unfortunately, I learned that our program has an emphasis on biology (and even if yours doesn't, I'm sure you'll still be exposed to some degree of it). This was a deal breaker; all that neuroscience, anatomy, and physiology bored me to tears, and I switched back to CS. :)