How to read large data set at hourly interval?

How can a physicist get into data science?

  • There's a huge demand for Data Scientists. I read about well-paying positions demanding multiple skills going unfilled due to not enough talented people to go around. I also read that physicists can do well as Data Scientists.  We're accustomed to dealing with complex systems, sophisticated statistics and abstract problem-solving.  We find patterns and signals in data that at first appears to be chaos and noise. Data Science appears to reside in the enterprise computing world of huge networks, transaction processing, The Cloud and massive data warehouses. A world filled with CS majors, finance and marketing people, and MBAs that I haven't yet set foot in.   My background is in physics, electronics, 2D and 3D graphics. I have worked with astronomers, electronics and mechanical engineers, and high energy physicists.  I write software and invent algorthms in the course of my work.  I also do a lot of writing and illustrating, answering physics and graphics questions, explaining test results and scientific phenomena to managers, executives and the public. Could I fill one of those hungry Data Science positions?  How does an experienced physicist/graphics/explainer get a foot in the door to start a new career in this field?  Will it pay well while I'm getting up to speed?  Could I be contributing useful algorithms, data visualizations or be solving major problems right away?  What kinds of organizations are most keen on hiring physicists to be Data Scientists?

  • Answer:

    Yes, but the term data science is kind of like the word engineer, t...

Stephen Ebbsky at Quora Visit the source

Was this solution helpful to you?

Other answers

I have a few suggestions, based on my experience: If you have or are finishing a PhD, consider doing the http://insightdatascience.com/. This is essentially a networking/job-placement program that could make your career switch very easy, but getting a slot in this program is competitive. If you can't get into (or don't want to do) the Insight Program, use the list of companies that they recruit for as a starting place for your job hunt. If your resume is less than stellar, look for positions at smaller and less prestigious companies. There is no shortage of data scientists applying to big name companies like Google or Facebook. Do an (paid) internship. It's a lot easier to get your foot in the door for a few months than to get a full time job offer. The internship also shows that you have invested some actual effort into your career path. Network. This one is pretty obvious, but the best way to get a job offer is through a referral. It's also a great way to find out about jobs you might not hear about otherwise. If looking for a place to learn on the job doesn't work, you may need to actually learn the skills on your own to prove your programing and statistics chops. Take the coursera course on machine learning. Do well in a Kaggle competition. Make an interesting big data demo (link to it on Hacker News). Put some code on GitHub. Contribute to an open source project. I did an internship at a late-stage tech startup in my last year of grad school and am now choosing between several job offers.

Stephan Hoyer

I've written a few blog posts about this topic.  I was an astrophysicist that has transitioned into data science in the last year: http://womeninastronomy.blogspot.com/2013/01/datascience.html http://womeninastronomy.blogspot.com/2013/01/astroVdatascience.html Hope these are helpful! Jessica

Jessica Kirkpatrick

First and foremost, I would go with what and mentioned. They give clear, solid advice: learn standard languages, learn about databases/data infrastructure, brush up on statistics, get as much experience as possible through internships or fellowships, and more. My suggestion, which could easily be another bullet on Stephan's list, would be to join a big company that has a data analytics team. I've read from some sources -- I forget where but I generally agree with this -- that you'll learn a whole lot more as a novice at a big company that already has a data analytics team rather than at a small startup where you are the head data scientist/expert. With a data analytics team in place there will not only be a lot of data to wrestle with but a lot of partners you can "tag-team" the data with to force it into submission.

Ming Law

You can try your hand at http://Kaggle.com competitions in your spare time (though I have heard that the competitions' regularly updated leaderboards have a way to really engage the competitors much more, especially if one is doing reasonably well.)   Consistently doing well in these could potentially open up doors.   Lots of press on Kaggle over the last 1-2 years -- will enable to you get more background on their competitions.

Mahesh M. Bhatia

I was A2A-ed here, but this question already has several great answers. Of the recommendations already given, I'd like to particularly highlight one: actually getting your hands dirty with a few data science projects. Ideally, you would do these using common tools of the trade like Python/R and SQL. I think this would have the highest ROI for handling data science interviews, while also giving you something to point to as evidence of your ability to generate results. I'll try to add a few suggestions that I haven't seen emphasized in the other answers: Explore machine learning: in contrast to other data science fundamentals (e.g., basic statistics, experimental design, programming), machine learning doesn't come up as frequently in physics research. You should get familiar with common classes of models, various error-rate metrics, cross-validation procedures, and potential pitfalls (e.g., over- and underfitting). already alluded to this in his (excellent) answer, but Andrew Ng's https://www.coursera.org/course/ml is a great place to start. Understand why you are making the transition to data science: You will almost certainly be asked why you're switching fields, and it helps to have a compelling and sincere reason. You should be able to explain that reason clearly. Be familiar with and enthusiastic about the product that you'll be working on: Companies like it when you know something about them and their mission. Additionally, if you use a product, you will have stronger intuitions about it; these intuitions will guide your work, and consequently, you will be more valuable to the company. Understand the types of metrics that people care about in your target industry: The metrics that I spend my days thinking about now are naturally very different from the ones that mattered for studying Anderson localization. I didn't do enough of this before interviewing, but it would have been helpful to read up on metrics that companies use to track their progress and the types of factors that can drive fluctuations in these metrics. Don't get discouraged by rejection: There are a lot of opportunities in this field at the moment, but data science means very different things at different companies, and not all opportunities will match your skillset. For example, some very small companies might not have the time to invest in helping a physicist transition to a new field. They may need experienced people who require little to no ramp up time. Nevertheless, if you are vigilant about opportunities, persistent in pursuing them, and learn from the interview process itself, there's a very good chance that you'll land somewhere exciting.

Shankar Iyer

Now is a great time to make the transition from Physics to Data Science.  The Insight Data Science Fellows Program seems nice, but don't count on it.  It is highly selective, with nearly all of the fellows coming from Ivy League (or close) schools, and my fear is that potential data scientists apply to that program and wait rather than working on more surefire ways to become a data scientist. My advice for making the transition from Physics to Data Science: Your experience is probably closer to Data Science than you realize, with the skills you've learned covering many of the core competencies that you will need. Each field of Physics and each project has its own strengths and experiences to offer.  As you learn more about Data Science, and the specifics of what skills and experiences are valued, you will be able to fine-tune your resume into something that will land you multiple interviews. Brush up on your programming.  This includes going back and learning more of the fundamentals, and working through practice problems.  You WILL have a programming section (or several) during each interview process, and you need to have the mindset to tackle both take-home problems as well as white-board pseudo-code. Network.  Talk to your classmates and coworkers.  Find out if anyone from your school has made the transition, and try to get advice from them or possible connections.  Alumni networks are bigger than you think, and those associated with major Astro or HEP experiments are even bigger, and those from national labs larger still. Learn the tools of the trade.  Download R or Julia.  Play around with the the Pandas and scikit-learn packages in Python.  Work through a sample Hadoop Streaming problem so you at least know the basics of Map/Reduce.  A little bit of tinkering goes a long way toward talking-the-talk. Compete in a Kaggle competition.  A good showing looks wonderful on the resume, but even if your results are not competitive it will give you valuable experience and will help give you perspective on some of the types of data problems and processes that you may face in a Data Science career. If you have a PhD in Physics or have progress toward one, then I don't think an internship is necessary.   If you're coming straight from undergrad, then an internship will be highly valuable. Sign up for GitHub and either contribute to projects or post your own work.  Your GitHub account is equivalent to an artist's, designer's, or writer's portfolio. There are plenty of Data Science jobs out there that are good for a Physicists, and anyone can get an offer somewhere if they try hard enough, but that doesn't mean you can float by without doing your due diligence.  Learn about the job, the field, the company, and know how your extensive research makes you qualified for that specific job.

Chris Prokop

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.