Why my SQLite3 query takes time?

SQLite3 query takes a lot of time, how would you do it?

  • I have a sqlite3 DB table of around 250 000 rows, my code is written in python. I need to filter it in very specific wat, and it takes much too long time. Table is as follows: self.cur.execute("""create table DetectedVehicles(IdD INTEGER PRIMARY KEY, CLCode INT, DetectionTime INT, PlateNo VARCHAR)""") It's a table for automatic plate number recognition result filtering. And I need to filter it to get (native sql-like statements :) ): Get rows from table DetectedVehicles where vehicles were observed at CLCode="X" before they were observed at CLCode="Y". (implicite: they were observed at both of them) So I need to get list of detectedvehicles, that crossed specific CLCodes in proper sequence, i.e. Y before X. I managed to create something that is working, but it takes about 10seconds for the query. Is there a faster way? The code goes here: self.cur.execute('select distinct PlateNo from DetectedVehicles where CLCode=? intersect select PlateNo from DetectedVehicles where CLCode=?',(CountLocationNo[0],CountLocationNo[1])) PlatesTab=list(self.cur) Results=[] for Plate in PlatesTab: PlateQ1='select * from DetectedVehicles where PlateNo in (?) and ((select DetectionTime from DetectedVehicles where CLCode = ? and PlateNo in (?) ) < (select DetectionTime from DetectedVehicles where CLCode = ? and PlateNo in (?)))' R=list(self.cur.execute(PlateQ1,(Plate,CountLocationNo[0],Plate,CountLocationNo[1],Plate))) if R: TimesOD=self.curST2.execute('select DetectionTime from DetectedVehicles where PlateNo in (?) and (CLCode= ? or CLCode=?)',(Plate,CountLocationNo[0],CountLocationNo[1])).fetchall() if TimesOD: TravelTimes.append(TimesOD[1][0]-TimesOD[0][0]) DetectionTimes.append(TimesOD[0][0]) for i in R: Results.append(i[0]) Results=tuple(Results) QueryCL=' intersect select * from DetectedVehicles where IDd in ' + str(Results) Thanks in advance

  • Answer:

    Try: select distinct x.* from DetectedVehicles x join DetectedVehicles y on x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime where x.CLCode=? and y.CLCode=? or: select x.* from DetectedVehicles x where exists (select 1 from DetectedVehicles y where x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime and x.CLCode=? and y.CLCode=?) I would normally expect the latter query to execute more quickly, but it would be worth running both to check.

Intelligent-Infrastructure at Stack Overflow Visit the source

Was this solution helpful to you?

Other answers

You can do it all in a single query. select dv1.PlateNo, dvPoint1.DetectionTime, dvPoint2.DetectionTime from DetectedVehicles dvPoint1 inner join DetectedVehicles dvPoint2 on dvPoint1.PlateNo = dvPoint2.PlateNo and dvPoint1.CLCode = ? and dvPoint2.CLCode = ? and dvPoint1.DetectionTime < dvPoint2.DetectionTime You will want an index on (PlateNo, DetectionTime, CLCode), or (CLCode, PlateNo). Try them both to see which is faster. PlateNo on it's own may do.

Ben

Thank You guys for this feedback. I post it as an answer, and present time results: 1. fastest total (query 1.80s, fetchall 0.20s, total: 2s) select distinct x.* from DetectedVehicles x join DetectedVehicles y on x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime where x.CLCode=? and y.CLCode=? 2. (query 1.83s, fetchall 0.19s, total: 2.02s) select dvPoint1.PlateNo, dvPoint1.DetectionTime, dvPoint2.DetectionTime from DetectedVehicles dvPoint1 inner join DetectedVehicles dvPoint2 on dvPoint1.PlateNo = dvPoint2.PlateNo and dvPoint1.CLCode = ? and dvPoint2.CLCode = ? and dvPoint1.DetectionTime < dvPoint2.DetectionTime 3. (query 1.82s, fetchall 1.09s, total: 2.91s) select x.* from DetectedVehicles x where exists (select 1 from DetectedVehicles y where x.PlateNo = y.PlateNo and x.DetectionTime < y.DetectionTime and x.CLCode=? and y.CLCode=?) So thanks @Mark Bannister for Your answer, I'm going to accept it. However one issue remains: cur.fetchall() takes enourmously long time.. and I need to get the results, how should I do it ? (For just a 100 of rows it takes around 2 minutes for each of your solutions). Solved issue: download new sqlite.dll to your python/dlls folder ... don't ask me why: http://stackoverflow.com/questions/3134900/join-with-pythons-sqlite-module-is-slower-than-doing-it-manually

Intelligent-Infrastructure

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.