SQLite3 query takes a lot of time, how would you do it?
-
I have a sqlite3 DB table of around 250 000 rows, my code is written in python. I need to filter it in very specific wat, and it takes much too long time. Table is as follows: self.cur.execute("""create table DetectedVehicles(IdD INTEGER PRIMARY KEY, CLCode INT, DetectionTime INT, PlateNo VARCHAR)""") It's a table for automatic plate number recognition result filtering. And I need to filter it to get (native sql-like statements :) ): Get rows from table DetectedVehicles where vehicles were observed at CLCode="X" before they were observed at CLCode="Y". (implicite: they were observed at both of them) So I need to get list of detectedvehicles, that crossed specific CLCodes in proper sequence, i.e. Y before X. I managed to create something that is working, but it takes about 10seconds for the query. Is there a faster way? The code goes here: self.cur.execute('select distinct PlateNo from DetectedVehicles where CLCode=? intersect select PlateNo from DetectedVehicles where CLCode=?',(CountLocationNo[0],CountLocationNo[1])) PlatesTab=list(self.cur) Results=[] for Plate in PlatesTab: PlateQ1='select * from DetectedVehicles where PlateNo in (?) and ((select DetectionTime from DetectedVehicles where CLCode = ? and PlateNo in (?) ) < (select DetectionTime from DetectedVehicles where CLCode = ? and PlateNo in (?)))' R=list(self.cur.execute(PlateQ1,(Plate,CountLocationNo[0],Plate,CountLocationNo[1],Plate))) if R: TimesOD=self.curST2.execute('select DetectionTime from DetectedVehicles where PlateNo in (?) and (CLCode= ? or CLCode=?)',(Plate,CountLocationNo[0],CountLocationNo[1])).fetchall() if TimesOD: TravelTimes.append(TimesOD[1][0]-TimesOD[0][0]) DetectionTimes.append(TimesOD[0][0]) for i in R: Results.append(i[0]) Results=tuple(Results) QueryCL=' intersect select * from DetectedVehicles where IDd in ' + str(Results) Thanks in advance
-
Answer:
Try: select distinct x.* from DetectedVehicles x join DetectedVehicles y on x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime where x.CLCode=? and y.CLCode=? or: select x.* from DetectedVehicles x where exists (select 1 from DetectedVehicles y where x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime and x.CLCode=? and y.CLCode=?) I would normally expect the latter query to execute more quickly, but it would be worth running both to check.
Intelligent-Infrastructure at Stack Overflow Visit the source
Other answers
You can do it all in a single query. select dv1.PlateNo, dvPoint1.DetectionTime, dvPoint2.DetectionTime from DetectedVehicles dvPoint1 inner join DetectedVehicles dvPoint2 on dvPoint1.PlateNo = dvPoint2.PlateNo and dvPoint1.CLCode = ? and dvPoint2.CLCode = ? and dvPoint1.DetectionTime < dvPoint2.DetectionTime You will want an index on (PlateNo, DetectionTime, CLCode), or (CLCode, PlateNo). Try them both to see which is faster. PlateNo on it's own may do.
Ben
Thank You guys for this feedback. I post it as an answer, and present time results: 1. fastest total (query 1.80s, fetchall 0.20s, total: 2s) select distinct x.* from DetectedVehicles x join DetectedVehicles y on x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime where x.CLCode=? and y.CLCode=? 2. (query 1.83s, fetchall 0.19s, total: 2.02s) select dvPoint1.PlateNo, dvPoint1.DetectionTime, dvPoint2.DetectionTime from DetectedVehicles dvPoint1 inner join DetectedVehicles dvPoint2 on dvPoint1.PlateNo = dvPoint2.PlateNo and dvPoint1.CLCode = ? and dvPoint2.CLCode = ? and dvPoint1.DetectionTime < dvPoint2.DetectionTime 3. (query 1.82s, fetchall 1.09s, total: 2.91s) select x.* from DetectedVehicles x where exists (select 1 from DetectedVehicles y where x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime and x.CLCode=? and y.CLCode=?) So thanks @Mark Bannister for Your answer, I'm going to accept it. However one issue remains: cur.fetchall() takes enourmously long time.. and I need to get the results, how should I do it ? (For just a 100 of rows it takes around 2 minutes for each of your solutions). Solved issue: download new sqlite.dll to your python/dlls folder ... don't ask me why: http://stackoverflow.com/questions/3134900/join-with-pythons-sqlite-module-is-slower-than-doing-it-manually
Intelligent-Infrastructure
Related Q & A:
- How can store sqlplus query in a file in formatted manner and display it to console?Best solution by Stack Overflow
- How much acreage is in a lot?Best solution by answers.com
- How do you pay for college if you have a lot of other bills?Best solution by wikihow.com
- How do you stop yourself from sweating a lot?Best solution by Yahoo! Answers
- How hard is it making a career out of sports betting?do a lot of people manage okay and make a decent salary?Best solution by Yahoo! Answers
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.